Exascale Computing Project Update

ECP Update
Douglas B. Kothe (ORNL), ECP Director
HPC User Forum
Argonne National Laboratory
Sep 10, 2019

2
Today’s Update
• Reminder of who we are
• BLUF
• Detailed plan for 2019 – 2023 now in place (ECP’s Final Design)
• Quick look in the review mirror
• Apps – great progress albeit with computational hurdles that are clearer now
• Our approach of developing/deploying co-designed computational motifs is working (exemplars)
• Crucial to ECP’s success is integration within (apps – S/W) and external to DOE HPC facilities
• A look ahead

3
US DOE Office of Science (SC) and National
Nuclear Security Administration (NNSA)
DOE Exascale Program: The Exascale Computing Initiative (ECI)
ECI
partners
Accelerate R&D, acquisition, and deployment to
deliver exascale computing capability to DOE
national labs by the early- to mid-2020s
ECI
mission
Delivery of an enduring and capable exascale
computing capability for use by a wide range
of applications of importance to DOE and the US
ECI
focus
Exascale
Computing
Project
(ECP)
Exascale system
procurement projects &
facilities
ALCF-3 (Aurora)
OLCF-5 (Frontier)
ASC ATS-4 (El Capitan)
Selected program
office application
development
(BER, BES,
NNSA)
Three Major Components of the ECI

4
Exascale Computing
in the United States
Douglas Kothe
Oak Ridge National Laboratory
Stephen Lee
Los Alamos National Laboratory
Irene Qualters
National Science Foundation
Abstract—The U.S. is a long-time international leader in HPC, rooted in a strong and
innovative computing industry that is complemented by partnerships with and among federal
agencies, academia, and industries whose success relies on HPC. The advent of exascale
computing brings challenges in traditional simulation as well as in areas colloquially referred
to as “Big Data.” Within this context, we describe the U.S. exascale computing strategy:
1) the National Strategic Computing Initiative, a multiple U.S. federal agency effort
comprehensively addressing computing and computational science requirements in the U.S.;
2) the Exascale Computing Initiative, a DOE effort to acquire, develop, and deploy exascale
computing platforms within DOE laboratories on a given timeline; and, 3) the Exascale
Computing Project (a component of the Exascale Computing Initiative), dedicated to the
creation and enhancement of applications, software, and hardware technologies for
exascale computers, focused on vital U.S. national security and science needs.
& THE U.S. IS a long-time international leader in
the research, development, and use of high-perfor-
mance computing (HPC). This leadership is rooted
in a strong and innovative U.S. computing industry
that is complemented by a variety of partnerships
with and among federal agencies, academia, and
industries whose own success relies on the capa-
bilities that HPC uniquely provides. HPC is a requi-
site for advanced modeling and simulation. Thus,
investments in new software and application
methods, models, and algorithms nationally and
internationally have accompanied technology
innovations across the HPC landscape. As higher
fidelity computational models have become possi-
ble, and even practical, the field is challenged
by not only the need for increases in computing
performance for more predictive models but by
fundamentally new opportunities arising from
“Big Data” in the form of new streaming instru-
ments and sensors, artificial intelligence and new
data analytics capabilities, and a growing interna-
tional demand for a highly skilled workforce.
Continued U.S. leadership and innovation
in HPC is essential to our economic, energy,
and national security. Large-scale HPC-based
Digital Object Identifier 10.1109/MCSE.2018.2875366
Date of publication 9 November 2018; date of current version
6 March 2019.
January/February 2019 1521-9615 ß 2018 IEEE
17
Table 1. ECP AD summary by Level 3 area, science challenges, and capabilities enabled by exascale computing.
ECP Level 3 Area Selected Science Challenges Selected Exascale-enabled Capabilities
Chemistry and Materi-
als Applications:
Understand underlying
properties of matter
needed to optimize and
control the design of
new materials and
energy technologies.
Materials discovery and design: Understanding and
control of material properties across different
length, time, and energy scales.
Discovery of new materials with desired
properties by combining computation, theory,
characterization, and synthesis.
High-performance simulations of complex
materials and phenomena, including point, line,
and planar defects; heterogeneity; interfaces; and
dynamic behavior, to simulate electricity storage
materials systems and phenomena at atomic and
molecular levels.
Multiscale (atoms-to-devices) predictive
simulations of materials, interfaces, complete
cells, and commercial-scale battery systems for
performance.
Creating optimized materials that possess a large
cross section for photo-absorption in the design
of improved photovoltaics.
Additive manufacturing: advanced prediction and
qualification of physical structures created
through additive manufacturing processes,
including material properties, laser melting,
solidification, etc.
Microscale modeling and analysis of grain and
solidification properties of various additive
manufacturing processes for material
performance prediction.
Chemical catalysis: rational design of chemical
catalysts
End-to-end, system-level descriptions of
multifunctional catalysis. Design of catalytic mate-
rials for energy production and manufacturing
Energy Applications:
Model existing and
future technologies for
the efficient and
responsible production
of energy to meet the
growing needs of
the U.S
Nuclear energy: Allow safe increased fuel
utilization, power upgrades, and reactor life
extensions and design of new, safe, cost-effective
advanced and small reactors.
Resolution and enhanced numerical modeling of
particle transport, fluid dynamics, and fluid/
structure interactions; enhanced coupled
multiscale physics of complex nonlinear
engineering-scale systems.
Combustion science: Increase efficiency of gas
turbines by potentially 25–50% and lowering of
emissions from internal combustion engines.
Wind energy: Predict the complex flow physics
within a wind plant composed of 100s of multi-MW
wind turbines to advance the fundamental
understanding of the flow physics governing
whole wind plant performance. Revolutionize the
design and control of wind farms and predict the
response of wind farms to a wide range of
atmospheric conditions.
Validated, predictive simulation capability for the
design cycle, allowing industry to reduce the
development time for efficient engines and
turbines. Harden wind plant design and layout
against energy loss susceptibility; higher
penetration of wind energy.
Earth and Space Sci-
ence Applications:
Address fundamental
scientific questions
from the origin of the
universe and chemical
elements to planetary
processes and interac-
tions affecting life and
longevity. Forecast
water resources and
severe weather with
increased confidence;
address food supply
changes.
Earth system modeling: Dynamical ecological and
chemical evolution of the Earth system.
Impact assessments and adaptation strategies in
Earth system models (regional-scale modeling;
aerosol and atmospheric modeling; coupling of
multiscale models of land, ice, and atmosphere).
Full integration of human dimensions components
to allow exploration of socioeconomic consequen-
ces of adaptation and mitigation strategies.
Environmental modeling: Multiscale sub-surface
biogeochemical research and modeling in support
of carbon cycling, environmental remediation,
and related activities.
Enhanced modeling of multiscale sub-terrain
structure/fluid interactions for a variety of
applications, including engineered carbon
sequestration solutions, hydraulic fracking
impacts, and plume modeling and analysis.
Race to Exascale
22 Computing in Science & EngineeringPublished by the IEEE Computer Society
space and other federal agencies’ mission
space, the ECP will create and enhance the
predictive capability of relevant applications
through targeted development of require-
ments-based models, algorithms, and methods
along with the development and integration of
required software in support of application
workflows.
As an example of how the success of ECP will
be measured at its conclusion, as well as the
technical challenges to be addressed, applica-
tions are involved in at least two of the KPPs
mentioned above, which can be generally con-
sidered as a measure of the performance of an
application on an exascale system compared to
today’s systems and the readiness of an applica-
tion for exascale-class computing.
A key concept of one of the KPPs is an applica-
tion performance baseline that is a quantitative
measure of application performance using the
fastest pre-exascale computers available today.
That is, each application targeting this KPP must
define a Figure of Merit (FOM) that meaningfully
represents the rate of “science work” for a given
challenge problem for a given application. Armed
with this baseline, this KPP is then measured by
taking the ratio of the performance of that appli-
cation on an exascale platform to one of the cho-
sen baseline pre-exascale systems. Examples of
FOMs range from a simple measure of speedup
for a fixed-size calculation (like particles per sec-
ond or grid cell updates per second) to a weak-
scaling measurement incorporating problem size
and performance (achieving the same calculation
Data Analytics and
Optimization
Applications:
Applications partially
based on modern data
analysis and machine
learning techniques
rather than strictly on
approximate solutions
to equations that state
fundamental physical
principles or reduced
semi-empirical models.
Health care: Application of advanced data
analytics and machine learning methods to the
identification and treatment of cancers.
The development of enhanced cancer detection
and identification methods, targeted for precision
medical treatment options and tools.
Phenotype from genotype: Rapid understanding of
the capabilities and vulnerabilities of new
organisms, e.g., for energy production and
disease control.
Draft cellular system model of a new microbe or
soil consortium in a week. High-throughput
genome annotation of eukaryotes in genome
sequencing.
Energy grid: Stabilizing and modernizing the
energy grid with dynamic power sources and
enhanced energy storage capabilities.
Optimization and stabilization of the energy grid
while introducing renewable energy sources.
More realistic decisions based on available energy
sources. Nonlinear feedback of consumers and
energy suppliers to deliver more reliable energy.
Enhanced power grid modeling, resiliency, and
cyber security to advance US energy security.
National Security
Applications:
Stewardship of the U.S.
nuclear stockpile and
related physics and
engineering modeling
and scientific inquiries
driven by the
stewardship mission.
Stockpile stewardship: Certification and
assessments to ensure that the nation’s nuclear
stockpile is safe, secure, and reliable.
Enhanced scientific understanding of key physical
phenomena through improved numerical
methods and spatial/temporal resolutions,
allowing for the removal of current key
approximations.
Enhanced quantification of margins and
uncertainties.
Enhanced resolution (spatial and temporal) and
accuracy of coupled multiscale, multi-physics
simulations.
Co-design Centers:
Integrate the rapidly
developing software
stack with emerging
hardware technologies,
while developing soft-
ware components that
embody the most com-
mon application motifs
to be integrated into the
respective application
software environments
for testing, use, and
requirements feedback.
Crosscutting algorithmic methods that capture
the most common patterns of computation and
communication in ECP applications.
Computational motifs included are structured
and unstructured grids (with adaptive mesh
refinement), dense and sparse linear algebra,
spectral methods, particle methods, Monte
Carlo methods, backtrack/brand-and-bound,
combinatorial logic, dynamic programming,
finite state machine, graphical models, graph
traversal, map reduce.
January/February 2019
23
Exascale Computing in the United States, Computing in Science and Engineering 21(1), 17-29 (2019)

5
ECP’s three technical areas have the necessary components to
meet national goals
Application
Development (AD)
Software
Technology (ST)
Hardware
and Integration (HI)
Performant mission and science applications @ scale
Aggressive RD&D
Project
Mission apps &
integrated S/W stack
Deployment to DOE
HPC Facilities
Hardware tech
advances
Integrated delivery of ECP
products on targeted systems at
leading DOE HPC facilities
6 US HPC vendors focused on
exascale node and system
design; application integration
and software deployment to
facilities
Deliver expanded and vertically
integrated software stack to
achieve full potential of exascale
computing
67 unique software products
spanning programming models
and run times, math libraries,
data and visualization
Develop and enhance the
predictive capability of
applications critical to the DOE
24 applications including
national security, to energy, earth
systems, economic security,
materials, and data

6
Software Technology
Mike Heroux, SNL
Director
Jonathan Carter, LBNL
Deputy Director
Hardware & Integration
Terri Quinn, LLNL
Director
Susan Coghlan, ANL
Deputy Director
Application
Development
Andrew Siegel, ANL
Director
Erik Draeger, LLNL
Deputy Director
Project Management
Kathlyn Boudwin, ORNL
Director
Manuel Vigil, LANL
Deputy Director
Doug Collins, ORNL
Associate Director
Al Geist, ORNL
Chief Technology Officer
Exascale Computing Project
Doug Kothe, ORNL
Project Director
Lori Diachin, LLNL
Deputy Project Director
Project Office Support
Megan Fielden, Human Resources
Willy Besancenez, Procurement
Sam Howard, Export Control Analyst
Mike Hulsey, Business Management
Kim Milburn, Finance Officer
Susan Ochs, Partnerships
Michael Johnson, Legal
and Points of Contacts at the
Core Laboratories
Julia White, ORNL
Technical Operations
Mike Bernhardt, ORNL
Communications
Doug Collins
IT & Quality
Monty Middlebrook
Project Controls & Risk
Industry Council
Dave Kepczynski, GE, Chair
Core Laboratories
Board of Directors
Bill Goldstein, Chair (Director, LLNL)
Thomas Zacharia, Vice Chair (Director, ORNL)
Laboratory Operations Task
Force (LOTF)
DOE HPC Facilities
ECP Organization
Dan Hoag
Federal Project Director
Barb Helland
ASCR Program Manager
Thuc Hoang
ASC Program Manager

7
ECP BLUF (Bottom Line Up Front)
• Recent (Jun 2019) external review of ECP’s “Final Design” reaffirmed that ECP is on track
– Result of comprehensive scrutiny of project structure, technical plans, and management processes
• ECP committed to detailed 2019-2023 technical plans and quantitative performance metrics (baseline)
• First-mover exascale system (Aurora, Frontier, El Capitan) schedules and technology targets set
• Formal relationships and shared-milestone plans with DOE HPC Facilities defined for mutual success
• Many unknown unknowns retired. Known unknowns remain, e.g., those related to robust, portable, and
performant accelerator programming model (Kokkos / Raja development and adoption growing)
• ECP’s AI/ML scope is cutting-edge & impactful (CANDLE, ExaLearn) - will expand as risks are retired
• Hardware and Integration (HI): PathForward element paying dividends; turning focus to deploying ECP’s E4S
(Extreme-Scale Scientific Software Stack); tuning applications to exascale systems
• Software Technology (ST): Deploying E4S now; defining ST product integration metrics; increasing focus on
software abstraction layers, hardware-driven algorithms for math libs (mixed precision), programming models
• Application Development (AD): Get skin in the game (quantitative criteria for challenge problems); refine plans
for exploitation of accelerators; what are the performance bottlenecks to delivering on the challenge problems?

8
ECP: From inception to present
• Status
Independent
Project Review
(IPR)
• Revised
Preliminary
Design Report
(Oct 2018)
• Final Design
Review of Final
Design Report
(Jun 2019)
FY19
• Status
Independent
Project Review
(IPR)
• Performance
Measurement Plan
Review
• Independent
Design Review of
Revised
Conceptual
Design Report
(Dec 2017)
FY18
• CD-1/3A mini-
review
• CD-1/3A
approval
• Revised
Conceptual
Design Report
(Apr 2017)
FY17
• AD and ST initial
scope selection
• Mission Need
Statement and CD-
0 approval
• Exascale
requirements Town
hall meetings
• Independent Cost
Review
• Independent Design
Review
• CD-1/3A Review
• Independent
Design Review of
Conceptual Design
Report (Mar 2016)
FY16
• Establish Project
Office
• Assemble
leadership team
• Initial project
structure
• Begin crafting
Mission Need
statement
• Exascale
requirements Town
hall meetings
• Exascale application
RFI issued to DOE
labs
• Draft ECI
Conceptual Design
Report (Sep 2015)
FY15

9
ECP’s final design has three primary components
Project Structure
• Three technical focus areas teamed
with project management expertise
• Hierarchical break down of work
scope with strong technical
management at each level
• Key Performance Parameters
(KPPs) to measure success in
meeting project objectives
• Critical dependencies
– Integration within the project
– Integration with DOE Facilities
Technical Plans
• Detailed definition of KPPs for
each project with minimal,
verifiable completion criteria
• Capability development plans for
each subproject including scope
and schedule
– Mileposts, milestones
• Technical risks and mitigation
strategies identified
• Key integration points and
dependencies identified
Management Processes
• Project planning
– Activity/milestone development
– Maintaining agility
• Project tracking
– Technical leaders and supporting
tools (Jira, Confluence, Primavera);
Dashboards; Milestone reports;
Monthly reports
• Project assessment
– External reviews; Milestone review
and approval; Stakeholder
discussions
• Dependency management
• Risk management
• Change management
Lori Diachin (LLNL, ECP Deputy)
lead this process to a successful
final design and review outcome

10
ECP applications target national problems in DOE mission areas
Health care
Accelerate
and translate
cancer research
(partnership with NIH)
Energy security
Turbine wind plant
efficiency
Design and
commercialization
of SMRs
Nuclear fission
and fusion reactor
materials design
Subsurface use
for carbon capture,
petroleum extraction,
waste disposal
High-efficiency,
low-emission
combustion engine
and gas turbine
design
Scale up of clean
fossil fuel
combustion
Biofuel catalyst
design
National security
Next-generation,
stockpile
stewardship codes
Reentry-vehicle-
environment
simulation
Multi-physics science
simulations of high-
energy density
physics conditions
Economic security
Additive
manufacturing
of qualifiable
metal parts
Reliable and
efficient planning
of the power grid
Seismic hazard
risk assessment
Earth system
Accurate regional
impact assessments
in Earth system
models
Stress-resistant crop
analysis and catalytic
conversion
of biomass-derived
alcohols
Metagenomics
for analysis of
biogeochemical
cycles, climate
change,
environmental
remediation
Scientific discovery
Cosmological probe
of the standard model
of particle physics
Validate fundamental
laws of nature
Plasma wakefield
accelerator design
Light source-enabled
analysis of protein
and molecular
structure and design
Find, predict,
and control materials
and properties
Predict and control
magnetically
confined fusion
plasmas
Demystify origin of
chemical elements

11
ECP Apps: Delivering on Challenge Problems
Requires Overcoming Computational Hurdles
Domain Challenge Problem Computational Hurdles
Wind Energy Optimize 50-100 turbine wind farms Linear solvers; structured / unstructured overset meshes
Nuclear Energy Virtualize small & micro reactors Coupled CFD + Monte Carlo neutronics; MC on GPUs
Fossil Energy Burn fossil fuels cleanly with CLRs AMR + EB + DEM + multiphase incompressible CFD
Combustion Reactivity controlled compression ignition AMR + EB + CFD + LES/DNS + reactive chemistry
Accelerator Design TeV-class 100-1000X cheaper & smaller AMR on Maxwell’s equations + FFT linear solvers + PIC
Magnetic Fusion Coupled gyrokinetics for ITER in H-mode Coupled continuum delta-F + stochastic full-F gyrokinetics
Nuclear Physics:
Lattice QCD
Use correct light quark masses for first
principle light nuclei properties
Critical slowing down; strong scaling performance of MG-
preconditioned Krylov solvers
Chemistry Heterogeneous catalysis: MSN reactions HF + DFT + coupled cluster (CC) + fragmentation methods
Chemistry Catalytic conversion of biomass Hybrid DFT + CC; CC energy gradients
Extreme Materials Microstructure evolution in nuclear matls AMD via replica dynamics; OTF quantum-based potentials
Additive Manufacturing Born-qualified 3D printed metal alloys Coupled micro + meso + continuum; linear solvers
Quantum Materials Predict & control matls @ quantum level Parallel on-node performance of Markov-chain Monte Carlo
Astrophysics
Supernovae explosions & neutron star
mergers
AMR + nucleosynthesis + GR + neutrino transport

12
ECP Apps: Delivering on Challenge Problems
Requires Overcoming Computational Hurdles
Domain Challenge Problem Computational Hurdles
Cosmology
Extract “dark sector” physics from upcoming
cosmological surveys
AMR or particles (PIC & SPH); subgrid model accuracy;
insitu data analytics
Earthquakes Regional hazard and risk assessment Seismic wave propagation coupled to structural mechanics
Geoscience
Geomechanical and geochemical evolution of a
wellbore system at near-reservoir scale
Coupled AMR flow + transport + reactions to Lagrangian
mechanics and fracture
Earth System
Assess regional impacts of climate change on
the water cycle @ 5 SYPD
Viability of Multiscale Modeling Framework (MMF) approach
for cloud-resolving model; GPU port of radiation and ocean
Power Grid Efficient planning; underfrequency response
Parallel performance of nonlinear optimization based on
discrete algebraic equations and MIP
Cancer Research
Predictive preclinical models and accelerate
diagnostic and targeted therapy
Increasing accelerator utilization for model search;
exploiting reduced/mixed precision; preparing for any data
management or communication bottlenecks
Metagenomics
Discover, understand (find genes) and control
species in microbial communities
Efficient and performant implementation of UPC, UPC++,
GASNet; graph algorithms; SpGEMM performance
FEL Light Source
Light source-enabled analysis of protein and
molecular structure and design
Strong scaling (one event processed over many cores) of
compute-intensive algorithms (ray tracing, M-TIP) on
accelerators

13
Co-design Projects
Co-design helps to ensure that
applications effectively utilize
exascale systems
• Pull software and hardware
developments into applications
• Pushes application requirements
into software and hardware
RD&D
• Evolved from best practice
to an essential element
of the development cycle
CD Centers focus on a unique
collection of algorithmic motifs
invoked by ECP applications
• Motif: algorithmic method that
drives a common pattern of
computation and communication
• CD Centers must address all
high priority motifs used by ECP
applications, including the new
motifs associated with data
science applications
Efficient mechanism
for delivering next-generation
community products with broad
application impact
• Evaluate, deploy, and integrate
exascale hardware-aware
software designs and
technologies for key crosscutting
algorithmic motifs into
applications
ExaLearn
Machine
Learning
ExaGraph
Graph-based
algorithms
CEED
Finite element
discretization
AMReX
Block structured
AMR
COPA
Particles/mesh
methods
CODAR
Data and
workflows
• Co-design centers address computational motifs common to
multiple application projects

14
Co-design centers each impact multiple applications
Co-design Application
CEED
ExaSMR, LLNL NNSA App, ExaAM, ExaWind, Combustion-PELE, Subsurface, E3SM,
SNL NNSA App
AMReX ExaAM, Combustion-PELE, MFIX-Exa, WarpX, ExaStar, ExaSky
CoPA EXAALT, ExaAM, WDMApp, MFIX-Exa, WarpX, ExaSky, AMReX
CODAR WDMApp, CANDLE, NWChemEx, EXAALT, Combustion-PELE, ExaSky
ExaGraph ExaWind, ExaBiome, ExaSGD, SNL NNSA App
ExaLearn ExaSky, CANDLE, NWChemEx, ExaAM

15
ECP’s Co-Design Center for Particle Applications: CoPA
Goal: Develop algorithms and software for
particle methods,
Cross-cutting capabilities:
• Specialized solvers for quantum
molecular dynamics (Progress / BML).
• Performance-portable libraries for
classical particle methods in MD, PDE
(Cabana).
• FFT-based Poisson solvers for
long-range forces.
Technical approach:
• High-level C++ APIs, plus a Fortran interface (Cabana).
• Leverage existing / planned FFT software.
• Extensive use of miniapps / proxy apps as part of the development process.
PI: Sue Mniszewski (LANL)

16
CoPA Cabana: co-designed numerical recipes for particle methods
Cabana:
• is a software library for developing exascale applications that use
particle algorith
• contains general particle data msstructures and algorithms
implemented with those data structures
• provides a platform to develop and deploy advanced scalable and
portable methods for particle-based physics algorithms
• is designed for modern DOE HPC architectures and builds
directly on Kokkos
• is open source and distributed on GitHub
Core ECP stakeholders include projects with codes for molecular
dynamics (MD), N-body and smoothed particle hydrodynamics (SPH),
and various particle-in-cell (PIC) derivatives.
Kokkos (ST)
Cabana (CoPA)
XGC
(WDMApp)
ExaMPM
(ExaAM)
CPU GPUMIC
CabanaMD
CabanaPIC
Mini-apps Production ECP apps
ARM
Stuart Slattery (ORNL)

17
Cabana Fortran and XGC Interoperability Efforts
• Particle push using Cabana
developed based on XGC
electron push kernel
– First Cabana implementation
that uses and can be used by
Fortran codes
– Beginning of PIC algorithm library
supplied by XGC for use in
Cabana
– Allows Fortran codes like XGC to
utilize Cabana data structure
flexibility and algorithms for
optimal HPC performance
• In development: Unstructured grid
– Unstructured grid field-gather
algorithm used in XGC being
implemented for use by Cabana
e
-
e
-
e
-
e
-
! Update particle positions
particles%x = particles%x + dx
// Call basic XGC push kernel
Cabana::parallel_for(0,n_part,
Electron_push);
Cabana_interface.cpp
electron_push.F90
! Prototype Fortran program
call Cabana_interface(particles)
main.F90
Interoperability strategy includes:
1. Calling existing Fortran kernels
that operate directly on Cabana-
allocated memory inside of
parallel loops
2. Fortran calls to Cabana
algorithms (e.g. sort, halo
exchange, etc.) to manipulate
particle data
Stuart Slattery (ORNL)

18
ECP’s Co-Designed Motif Approach is Working
ECP’s CoPA is ensuring portable performance of the XGC fusion application
• XGC utilizes Cabana/Kokkos for portable platform performance: Summit, Perlmutter, Aurora,
Frontier, Post-K
• Fortran interface has been developed for XGC via Cabana particle library in ECP-CoPA
• Similar performance on Summit and KNL
Lowerisbetter
50M electrons/GPU
2.4T ions and electrons on
90% Summit
2.4T
Original version
Kokkos version
17 PI: Amitava Bhattacharjee (PPPL)

19
ExaLearn: Machine Learning for Inverse Problems in Materials Science
Sudip Seal (ORNL)

20
Machine Learning in the Light Source Workflow
Compressor
Nodes
Local SystemsBeam Line Control and
Data Acquisition (DAQ)
Network Remote Exascale HPC
TB/s
Exascale
Supercomputer
10 GB/s - 1Tb/s
Online
Monitoring and
Fast Feedback
ML for fast analysis
at the experimental
facility. Uses models
learned remotely.
ML to control
the beam line
parameters Simulate
experiments, beam
line control and
diffraction images at
scale to create data
for training
ML networks for image
classification, feature
detection and solving inverse
problems (how to change
experiment params to get
desired experiment result)
DAQ
Model
Model
Model
Model
Data Data Data Data Data
Model
Model
ML to design
light source
beam lines
ML at DAQ to
control data as
it is acquired
ML for data
compression
(e.g. hit finding).
Use models
learned remotely.
PI: Frank Alexander (BNL)

21
ECP Software Technology (ST)
Develop and deliver high-quality
and robust software products
Guide, and complement, and
integrate with vendor efforts
Prepare SW stack for scalability
with massive on-node parallelism
Extend existing capabilities when
possible, develop new when not
Goal
Build a comprehensive, coherent
software stack that enables
application developers to
productively write highly
parallel applications
that effectively target
diverse exascale
architectures

22
ECP ST Software Ecosystem
ECP Applications
Software Ecosystem & Delivery
Development
Tools
Programming
Models
Runtimes
Mathematical
Libraries
Data &
Visualization
Facilities Vendors HPC Community
ECP Software Technology
Collaborators (with ECP HI)
Details available publicly at https://www.exascaleproject.org/wp-content/uploads/2019/02/ECP-ST-CAR.pdf

23
Programming
Models & Runtimes
• Enhance & prepare
OpenMP and MPI
programming
models (hybrid
programming
models, deep
memory copies) for
exascale
• Development of
performance
portability tools (e.g.
Kokkos and Raja)
• Support alternate
models for potential
benefits and risk
mitigation: PGAS
(UPC++/GASNet)
,task-based models
(Legion, PaRSEC)
• Libraries for deep
memory hierarchy &
power management
Development
Tools
• Continued,
multifaceted
capabilities in
portable, open-
source LLVM
compiler
ecosystem to
support expected
ECP architectures,
including support
for F18
• Performance
analysis tools that
accommodate new
architectures,
programming
models, e.g., PAPI,
Tau
Math Libraries
• Linear algebra,
iterative linear
solvers, direct
linear solvers,
integrators and
nonlinear solvers,
optimization, FFTs,
etc
• Performance on
new node
architectures;
extreme strong
scalability
• Advanced
algorithms for multi-
physics, multiscale
simulation and
outer-loop analysis
• Increasing quality,
interoperability,
complementarity of
math libraries
Data and
Visualization
• I/O via the
HDF5 API
• Insightful,
memory-efficient
in-situ
visualization and
analysis – Data
reduction via
scientific data
compression
• Checkpoint restart
Software
Ecosystem
• Develop features in
Spack necessary to
support all ST
products in E4S, and
the AD projects that
adopt it
• Development of
Spack stacks for
reproducible turnkey
deployment of large
collections of
software
• Optimization and
interoperability of
containers on HPC
systems
• Regular E4S
releases of the ST
software stack and
SDKs with regular
integration of new
ST products
NNSA ST
• Open source NNSA
Software projects
• Projects that have
both mission role
and open science
role
• Major technical
areas: New
programming
abstractions, math
libraries, data and
viz libraries
• Cover most ST
technology areas
• Subject to the same
planning, reporting
and review
processes
ECP software technologies are a fundamental underpinning in
delivering on DOE’s exascale mission
10-8
10-4
10
0
10
4
0 100 200 300 400 500 600 700 800 900
Residual
Iteration
PAPI SDE Recorder: Residual per Iteration (662-bus: 662 x 662 with 2,474 nonzeros)
CG
CGS
BICGSTAB

24
Software Development Kits (SDKs): Key delivery vehicle for ECP
A collection of related software products (packages) where coordination across package teams improves usability
and practices, and foster community growth among teams that develop similar and complementary capabilities
• Domain scope
Collection makes functional sense
• Interaction model
How packages interact; compatible, complementary, interoperable
• Community policies
Value statements; serve as criteria for membership
• Meta-infrastructure
Invokes build of all packages (Spack), shared test suites
• Coordinated plans
Inter-package planning. Augments autonomous package planning
• Community outreach
Coordinated, combined tutorials, documentation, best practices
ECP ST SDKs: Grouping similar products
for collaboration & usability
Programming Models &
Runtimes Core
Tools & Technologies
Compilers & Support
Math Libraries (xSDK)
Viz Analysis and Reduction
Data mgmt., I/O Services & Checkpoint/
Restart
“Unity in essentials, otherwise diversity”

25
Extreme-scale Scientific Software Stack (E4S)
A Spack-based distribution of ECP ST products and related and dependent software tested for interoperability
and portability to multiple architectures
Lead: Sameer Shende, University of Oregon
• Provides distinction between SDK usability / general
quality / community and deployment / testing goals
• Will leverage and enhance SDK interoperability thrust
• Releases:
– Oct: E4S 0.1: 24 full, 24 partial release products
– Jan: E4S 0.2: 37 full, 10 partial release products
• Current primary focus: Facilities deployment
http://e4s.io

26
The ECP has several interdependencies both within the project and
with the DOE Facilities
DOE Facility Dependencies
• ECP requires access to Facility
resources to develop, test, and
demonstrate KPPs
• ECP software stack must
leverage and complement
vendor and Facility software
stack
• PathForward program
designed to keep US industry
healthy and feed into Facility
procurements
AD
• App team dependence on
co-design software and
tools
• App teams interacting
ST
• Integrated Software Stack
(SDKs, E4S)
• Programming models
used throughout
• Math library dependencies
• Use of development tools
for productivity
HI
• Hardware evaluation with
PathForward
• Joint surveys to determine
software stack at Facilities
AD/ST
• Strong dependence of apps
on ST tools and libraries
ST/HI
• Continuous integration
process for software
testing
• Spack package
management
AD/HI
• Application integration at
Facilities
• First movers program
AD/ST/HI
• App performance optimization
• Software stack determination for
Facilities
• Access to Facility resources
• Training and Productivity

27
Managing AD-ST complexity, i.e. “taming the hairball”
• We currently have significant usage of ST and co-design
products by AD application teams.
• To manage dependencies, it was necessary to first
gather accurate data:
– AD applications filled out detailed tables of software
specs and dependencies on Confluence
– ST teams reported application dependencies
– HI interviews with application teams
• Data was not initially fully consistent.
– ST teams reported working with applications who
didn't list them as dependencies
– Applications reported depending on ST projects who
didn't list them as customers
• Consistent interdependency data now being imported
into ECP’s database for configuration control, analysis
and planning
– Has this ever been done? We don’t think so . . . ST product
AD project

28
Examples of applications’ use of ST/CD products
Chemistry and Materials Applications
ExaAM / ALExa
• In-memory coupling framework
critical for multi-physics + multi-
scale efficiency at exascale
• ExaAM used DTK for coupling
Truchas and Diablo - the two
main components in AM
evolution
• Coupling key to two milestones
QMCPACK / SOLLVE+Kokkos
• Portable programming required
for CPU, GPU and other
accelerated systems
• High priority kernels ported to
both OpenMP and Kokkos and
their preliminary performance has
be assessed.
• OpenMP GPU branch provides
current best FOM on Summit
EXAALT / SLATE
• Fast quantum-based MD (QMD)
requires high performance linear
algebra subroutines
• UTK’s SLATE package provides
multi-GPU version of BLAS level 3
generalized matrix matrix
multiplication subroutine for high
performance QMD solvers
• LATTE QMD calls the new SLATE
DGEMM subroutine for generalized
matrix matrix multiplication through
a new, easy-to-use API
• LATTE runs with good performance
on CPUs or multiple GPUs by
changing an environment variable at
run time through a common
interface
ExaAM
ST
ALExa
Kokkos
xSDK
ADIOS

29
Energy Applications
ExaWind / Kokkos-Trilinos-Hypre
• Test/optimize multiple solver
solutions to meet strong-scaling
challenges from moving mesh
continuity/momentum solves
each timestep
• Multiple solver stacks integrated,
verified, and performance tested;
optimization ongoing
• Will overcome strong-scaling limit
for full windfarm modeling
• Provides performance feedback
to solver teams that will benefit
other projects
MFIX-Exa / AMReX
• Provide adaptive mesh
refinement, field solvers for
implicit projection Navier-Stokes,
and embedded boundaries for
non-orthogonal geometries
• Embedded boundaries and
implicit projection solver
integrated and tested;
performance optimization and
scaling ongoing
• Allow performance-portable DEM
mechanics on combinatorial
geometry reactor models
WDMApp / ADIOS
• Provide high-performance IO and
coupling framework for
Gyrokinetic codes
• ADIOS is the backbone of the
KITTIE framework that allows
coupling between GENE (core)
and XGC (edge) gyrokinetic
codes
• Will allow incorporation of
community-fusion models into
WDMApp for exascale whole
device modeling

30
Earth and Space Science Applications
EQSIM / RAJA
• Codes need to exploit
accelerators to meet the KPP
goals.
• The project opted to use RAJA
as the technology for using
accelerators.
• SW4 has been integrated with
RAJA, and the project team
exercised the code on Sierra.
The performance results were
excellent.
• This integration activity has
saved the project from having to
develop accelerator specific code
and allowed them to retain
performance portability.
E3SM-MMF / ADIOS
• The larger E3SM project has a
large code base with integration
with PIO1 for aggregation of data
and PnetCDF and HDF5 for I/O.
• The MMF code is integrated into
the E3SM-MMF code base.
• For the MMF, new integration
points are PIO2, and support for
ADIOS in PIO2.
• The E3SM-MMF project has
successful collaboration with both
the PIO2 and ADIOS projects.
ExaStar / AMReX
• Exastar is developing CLASH as
shared computing ecosystem for
astrophysics applications of
interest.
• Two primary codes FLASH and
Castro share common
capabilities through defined
interfaces, and both use AMReX
for mesh management.
• FLASH is newly integrated with
AMReX under ECP to replace
the unsupported Paramesh
library. Castro was built with
AMReX.
• Both codes also benefit from
other integration activities in
AMReX, such as interface with
PETSc and hypre.

31
Data Analytics and Optimization Applications
ExaBiome / ExaGraph
• The protein clustering HipMCL
code uses graph algorithms and
sparse matrix kernels from
GraphBLAS.
• HipMCL was greatly accelerated
using the shared-memory parallel
hash-based SpGEMM algorithm
and the distributed-memory
parallel linear-algebraic
connected components
algorithm.
• The HipMCL code made it
possible to cluster previous
intractable data sets with over
200 million proteins.
ExaFEL / NERSC
• A Software Defined Networking
(SDN) solution was used to
enable ExaFEL traffic from SLAC
to NERSC to selectively request
an uncongested path within
Esnet.
• This minimized impact on other
site traffic and provided network
performance predictability for the
transfer.
• ExaFEL developed an API for
network provisioning. This API
was deployed via an SDN data
plane in coordination with
NERSC staff.
CANDLE / ExaLearn
• The capabilities being developed
by CANDLE and ExaLearn are
highly synergistic.
• ExaLearn is exploiting CANDLE
capabilities for deep learning to
develop surrogate models of
exascale simulations.
• CANDLE provided a lot of
context for the later introduction
of ExaLearn, which helped
focused this project to ensure
effective impact on ECP
applications.
• CANDLE and ExaLearn share
key team members to facilitate
communication and coordination
between these teams.

32
ECP Hardware and Integration (HI)
Training on key ECP technologies, help in
accelerating the software development cycle
and in optimizing the productivity of application
and software developers
A well integrated and continuously tested
exascale software ecosystem deployed at DOE
facilities through collaboration with facilities
Innovative supercomputer architectures for
competitive exascale system designs
Accelerated application readiness through
collaboration with the facilities
Goal
A capable exascale computing
ecosystem made possible
by integrating ECP
applications, software
and hardware
innovations within
DOE facilities
Aurora
Access to the computer resources at facilities:
early access, test and dev. systems, and pre-
exascale and exascale systems

33
Application Matching to Facilities Plan and Status
NERSC
ALCF
OLCF
Goal: 21 performant
exascale applications that
run on Aurora and/or
Frontier
Strategy: Match applications
with existing facility readiness
efforts
Progress Assessment: Progress
towards technical execution plans
measured quarterly; annual
external assessment.
5 ECP AD applications participating
in NESAP for NERSC-9 . Additional
applications may participate with
NERSC funding.
Goal: Progress
towards exascale
readiness develops,
and NESAP-ECP
apps transition to
LCF facilities
12 initial applications engaged by ALCF for
Aurora. Other teams can follow best
practices for Aurora readiness, and will be
engaged as staffing allows.
An initial set of 12 ECP applications identified
to participate in CAAR-ECP in FY19.
Applications may transition in and out of the
program as progress is made.

34
ECP: The Road Ahead
• CD-4 Review and
Approval (project
completion)
• Deliver KPP
completion
evidence
• Access to Aurora
and Frontier full
system
FY23
• Status
Independent
Project Review
(IPR)
• AD and ST
readiness
demonstrated
• Access to earliest
Aurora and
Frontier full system
FY22
• Status
Independent
Project Review
(IPR)
• AD application
projections firm
for target system
• ST integration
goals assessed
• Access to Aurora
and Frontier Test
and Development
Systems (TDS)
FY21
• CD-2/3 Review and
Approval
• Did PathForward
deliver? Are AD and
ST performance and
integration
projections on
track?
• Access to early
hardware
FY20
• Final Design
Review
• Establish
performance
baseline
• AD KPP completion
criteria and ST
integration goals set
• Access to pre-
exascale systems
FY19

Exascale Computing Project Update

More Related Content

What's hot (20)

Similar to Exascale Computing Project Update (20)

More from inside-BigData.com (20)

Recently uploaded (20)

Exascale Computing Project Update