Sierra overview
- 1. LLNL-PRES-738369 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC Sierra: The LLNL IBM CORAL System Bronis R. de Supinski Chief Technology Officer Livermore Computing October 6, 2017
- 2. LLNL-PRES-738369 2 AdvancedTechnology Systems(ATS) Fiscal Year ‘13 ‘14 ‘15 ‘16 ‘17 ‘18 Use Retire ‘20‘19 ‘21 Commodity Technology Systems(CTS) Procure& Deploy Sequoia (LLNL) ATS 1 – Trinity (LANL/SNL) ATS 2 – Sierra (LLNL) Tri-lab Linux Capacity Cluster II (TLCC II) CTS 1 CTS 2 ‘22 System Delivery ATS 3 – Crossroads (LANL/SNL) ATS 4 – (LLNL) ‘23 ATS 5 – (LANL/SNL) Sierra will be the next ASC ATS platform Sequoia and Sierra are the current and next-generation Advanced Technology Systems at LLNL
- 3. LLNL-PRES-738369 3 Sierra is part of CORAL, the Collaboration of Oak Ridge, Argonne and Livermore CORAL is the next major phase in the U.S. Department of Energy’s scientific computing roadmap and path to exascale computing Modeled on successful LLNL/ANL/IBM Blue Gene partnership (Sequoia/Mira) Long-term contractual partnership with 2 vendors 2 awardees for 3 platform acquisition contracts 2 nonrecurring eng. contracts NRE contract ANL Aurora contract (2018 delivery) NRE contract ORNL Summit contract (2017 delivery) LLNL Sierra contract (2017 delivery) RFP BG/Q Sequoia BG/L BG/P Dawn LLNL’s IBM Blue Gene Systems
- 4. LLNL-PRES-738369 4 The Sierra system that will replace Sequoia features a GPU-accelerated architecture Mellanox Interconnect Single Plane EDR InfiniBand 2 to 1 Tapered Fat Tree IBM POWER9 • Gen2 NVLink NVIDIA Volta • 7 TFlop/s • HBM2 • Gen2 NVLink Components Compute Node 2 IBM POWER9 CPUs 4 NVIDIA Volta GPUs NVMe-compatible PCIe 1.6 TB SSD 256 GiB DDR4 16 GiB Globally addressable HBM2 associated with each GPU Coherent Shared Memory Compute Rack Standard 19” Warm water cooling Compute System 4320 nodes 1.29 PB Memory 240 Compute Racks 125 PFLOPS ~12 MW GPFS File System 154 PB usable storage 1.54 TB/s R/W bandwidth
- 5. LLNL-PRES-738369 5 Outstanding benchmark analysis by IBM and NVIDIA demonstrates the system’s usability Projections included code changes that showed tractable annotation-based approach (i.e., OpenMP) will be competitive Figure 5: CORAL benchmark projections show GPU-accelerated system is expected to deliver substan performance at the system level compared to CPU-only configuration. The demonstration of compelling, scalable performance at the system level across a 13 CORAL APPLICATION PERFORMANCE PROJECTIONS 0x 2x 4x 6x 8x 10x 12x 14x QBOX LSMS HACC NEKbone RelativePerformance Scalable Science Benchmarks CPU-only CPU + GPU 0x 2x 4x 6x 8x 10x 12x 14x CAM-SE UMT2013 AMG2013 MCB RelativePerformance Throughput Benchmarks CPU-only CPU + GPU
- 6. LLNL-PRES-738369 6 Sierra system architecture details have recently been finalized with Go decision Sierra uSierra Nodes 4,320 684 POWER9 processors per node 2 2 GV100 (Volta) GPUs per node 4 4 Node Peak (TFLOP/s) 29.1 29.1 System Peak (PFLOP/s) 125 19.9 Node Memory (GiB) 320 320 System Memory (PiB) 1.29 0.209 Interconnect 2x IB EDR 2x IB EDR Off-Node Aggregate b/w (GB/s) 45.5 45.5 Compute racks 240 38 Network and Infrastructure racks 13 4 Storage Racks 24 4 Total racks 277 46 Peak Power (MW) ~12 ~1.8 These are working numbers; the final configuration will only be set once the system is fully installed
- 7. LLNL-PRES-738369 7 LLNL and ASC platform chose to use a tapered fat-tree for Sierra’s network This decision, counter to prevailing wisdom for system design, benefits Sierra’s planned UQ workload: 5% more UQ simulations at performance loss of < 1% Full bandwidth from dual ported Mellanox EDR HCAs to TOR switches Half bandwidth from TOR switches to director switches An economic trade-off that provides approximately 5% more nodes
- 8. LLNL-PRES-738369 8 NVIDIA Volta GPUs (GV100) provide the bulk of Sierra’s compute capability To realize Sierra’s full potential, we must exploit the tensor operations. The commoditization of machine learning will make this an enduring challenge. 9 VOLTA GV100 SM GV100 FP32 units 64 FP64 units 32 INT32 units 64 Tensor Cores 8 Register File 256 KB Unified L1/Shared memory 128 KB Active Threads 2048 SMs 80 FP64 Units (per SM) 32 FP32 Units (per SM) 64 Tensor Cores (per SM) 8 Register File (per SM) 256KiB L1/Shared Memory (per SM) 128KiB Double Precision Peak (TFlop/s) 7 (7.5) Single Precision Peak (TFlop/s) 14 (15) Tensor Op Peak (TOp/s) 120 HBM2 Bandwidth (GB/s) 898 NVLINK BW to CPU/Other GPU 75 (60)
- 9. LLNL-PRES-738369 9 The advantages that led us to select Sierra generalize —Power efficiency —Network advantages of “fat nodes” —Balancing capabilities/costs implies complex memory hierarchies Planning a similar, unclassified, M&IC resource —Same architecture as Sierra —Up to 25% of Sierra’s capability Exploring possibilities for other GPU-based resources —Not necessarily NVIDIA-based —May support higher single precision performance Sierra and its EA systems are beginning an accelerator-based computing era at LLNL We have multiple projects planned to foster a healthy ecosystem