My interests are in high performance scientific computing, with
particular emphasis on parallel computing.
These include the development of scalable algorithms, programming
systems to enable the efficient and correct expression of algorithms,
and tools for writing, understanding, and tuning parallel
I am also involved in defining and providing the infrastructure for
high performance computing and for a large science project, the Large
Synoptic Survey Telescope.
A common underlying issue is matching the algorithms and software to
the computer architecture, particularly as architectures adapt,
possibly with radical innovations, to the
end of Dennard scaling (around 2004) and the end of Moore's Law (in
rigourous terms, also already ended, in practice, slowly fading since
it is really an engineering imperative rather than a natural law).
- Programming systems
- Effective ways to program parallel
computers require (a) a standard approach, (b) a high-quality,
ubiquitous implemenation of that standard, and (c) effective ways
to get high performance on each core and node.
- MPI Standard
The Message Passing
Interface (MPI) standard is the dominant programming system for highly
parallel HPC applications. I have been involved from the beginning
of MPI process and currently serve as the overall standard editor
and the co-chair of the RMA chapter.
- MPICH implementation
- The MPICH implementation of MPI is the
most widely used implementation of MPI. I began this
implementation while MPI was originally being defined, and it
continues with the major work at Argonne National Laboratory. My
recent contributions have included better implementations for
datatypes, remote memory access, collective communications, and
- Code transformations
- Parallelism is only one
approach to performance. Real HPC applications ofter require
non-trivial code transformations to go beyond what a compiler is
willing or able to perform. Recent projects in this area include
the Illinois Coding Environment (ICE) and a Just-In-Time (JIT)
compilation approach for HPC applications called Moya.
- Parallel I/O
- I/O in parallel applications often
performs spectacurally poorly. I have worked on measuring I/O
performance and establishing performance expectations,
contributing to the ROMIO implementation of MPI-IO, developing
application-level libraries (such as MeshIO) for better
performance, and exploring the possibilities for post POSIX I/O
specifications in HPC.
- Performance Models and Tools
- Quantifing performance
and developing performance models and especially performance
expectations. Recent work has included a new performance
model that better matches the performance of internode
communication than the classical postal model; this new model is
called the max rate model. Software to measure
communication performance include
nodecomm. FPMPI is a software
that uses the MPI profiling interface to gather details about the
use of MPI in an application without requiring any source code
is a collection of routines and
programs for parallel programs. Included in baseenv is a
comprehensive set of tests of compiler vectorization.
Some introductory papers on performance modeling
- Scalable Numerical Methods
PETSc is a suite of data
structures and routines for the scalable (parallel) solution of
scientific applications modeled by partial differential
- At extreme scale, synchronizing operations such as reductions
(dot products) can limit scalability. I have investigated ways to
use non-blocking collective operations in MPI to implement
scalable Krylov Methods.
Most of these grants are team projects in which my role runs from
being the principle investigator to one of several co-investigators.
In additional to grants of funds, I have several grants of
supercomputer time. These grants have significant value in dollars
but are rarely included in lists of grants because no funds are
exchanged (and for the institutions, there is no overhead collected).
- Center for the Exascale
Simulation of Plasma Coupled Combustion (XPACC), a >$20M, 6
year project in the DOE Predictive Science program.
NCSA leads the data facility for the LSST project.
- Blue Waters is
NSF's most powerful general purpose supercomputer, and has
sustained more than a petaFLOP on a wide variety of applications.
- Midwest Big Data Hub
(MBDH) is one of the four regional data hubs funded by the NSF
Learning Major Research Instrument is an instrument to support
deep learning researchers across the University of Illinois
Urbana-Champaign campus. This is a development MRI, in partnership
with IBM and NVIDIA
- Scalable solvers for reservoir simulation, a joint project with
Luke Olson and funded by ExxonMobil.
- Lattice QCP Exascale Computing Project, one of the DOE Exascale
- Scalable and Highly Accurate Methods for Metagenomics, led at
Illinois by Tandy Warnow
- While not a grant in the conventional sense, I administer a
Siebel Fellowship to a student at UIUC; I've used this for students
pursuing topics in big data and bioinformatics.
- As a Blue Waters Professor, I have a grant of time on the Blue
Waters system that I use for a variety of projects
- As part of the PEAC-INCITE award, I have access to time on
supercomputers operated by DOE's Office of Science.