Update on Trinity System Procurement and Plans
3 likes1,665 views
Trinity is an advanced technology supercomputer system for the ASC Program that will be located at Los Alamos National Laboratory. It is designed to support the largest and most demanding applications of the ASC Program. Trinity will have a peak performance of over 42 petaflops and will introduce new technologies like the Intel Knights Landing processor, burst buffer storage, and advanced power management. The system is scheduled to be delivered in September 2015 and will help the ASC Program prepare for future exascale systems.
Update on Trinity System Procurement and Plans
- 1. Trinity: Advanced Technology System for the ASC Program Manuel Vigil Trinity Project Director High Performance Computing Division Los Alamos National Laboratory HPC User Forum September 17, 2014 LA-‐UR-‐14-‐27024
- 2. NERSC-8 and Trinity team activities Market surveys NERSC-‐8 Trinity Market surveys Joint Market surveys Crea6ng requirements Release RFP Vendor Selec6on Some joint nego6a6ons Common base SOW language? Nego6a6ons Con$nue to work together for CORI and Trinity deployments Nego6a6ons Separate SOWs Separate SOWs
- 3. Topics • Trinity Status • ASC Compu@ng Strategy • Trinity Project Drivers & Mission Need • The Trinity System – High-‐level architecture overview – High-‐level capabili@es – Schedule • Summary 3
- 4. Trinity Status • Formal Design Review and Independent Project Review completed in 2013 • Trinity/NERSC8 RFP released August 2013 • Technical Evalua@on of the proposals completed September 2013 • Ini@al nego@a@ons for both systems completed November 2013 • Ini@al Trinity system procurement completed in late 2013/early 2014 – Before final approval Trinity went back to the proposing vendors for a Best and Final Offer (BAFO) – Target delivery date of September 2015 is unchanged • Trinity Best and Final Offer RFP released to vendors March 2014 • Trinity proposal evalua@ons and nego@a@ons completed April 2014 • Trinity Procurement Approval – No@ce of Consent from NNSA received May 16, 2014 • Trinity Independent Cost Review completed May 2014 • Trinity CD-‐2/3b approved July 3, 2014 • Trinity Contract Awarded to Cray, Inc. on July 9, 2014 4
- 5. ASC Compu;ng Strategy • Approach: Two classes of systems – Advanced Technology: First-of-a-kind systems that identify and foster technical capabilities and features that are beneficial to ASC applications – Commodity Technology: Robust, cost-effective systems to meet the day-to-day simulation workload needs of the program • Advanced Technology Systems – Leadership-class platforms – Pursue promising new technology paths – These systems are to meet unique mission needs and to help prepare the program for future system designs – Includes Non-Recurring Engineering (NRE) funding to enable delivery of leading-edge platforms – Acquire right-sized platforms to meet the mission needs – Trinity will be deployed by ACES (New Mexico Alliance for Computing at Extreme Scale, i.e., Los Alamos & Sandia) 5
- 6. Advanced Technology Systems (ATS) Cielo (LANL/SNL) Sequoia (LLNL) ‘12 ‘13 ‘14 ‘15 ‘16 ‘17 Fiscal Year Use Retire ‘18 ‘19 ‘20 Commodity Technology Systems (CTS) Dev. & Deploy ATS 1 – Trinity (LANL/SNL) ATS 2 – (LLNL) ATS 3 – (LANL/SNL) Tri-‐lab Linux Capacity Cluster II (TLCC II) CTS 1 CTS 2 ‘21 System Delivery ASC Plaborm Timeline ASC Compu@ng Strategy includes applica@on code transi@on for all plaborms 6
- 7. Trinity Project Drivers and Mission Need • Sa@sfy the mission need for more capable plaborms – Trinity is designed to support the largest, most demanding ASC applica@ons – Increases in geometric and physics fideli@es while sa@sfying analysts’ @me-‐to-‐ solu@on expecta@ons • Mission Need developed with tri-‐lab input • Trinity will support the tri-‐lab applica@ons community at LLNL, SNL, and LANL • Mission Need Requirements are primarily driving memory capacity – Over 2 PB of aggregate main memory – Trinity is sized to run several jobs using about 750 TBytes of memory 7
- 8. Overview of Trinity Award • Subcontractor – Cray, Inc. • Firm Fixed Price Subcontract: – Trinity Plaborm (including File System) – Burst Buffer – 2 Applica@on Regression Test Systems – 1 System Development Test System – On-‐site System and Applica@on Analysts – Center of Excellence for Applica@on Transi@on Support – Advanced Power Management – Trinity System Maintenance 8
- 9. Trinity PlaBorm • Trinity is a single system that contains both Intel Haswell and Knights Landing processors – Haswell par@@on sa@sfies FY15 mission needs (well suited to exis@ng codes). – KNL par@@on delivered in FY16 results in a system significantly more capable than current plaborms and provides the applica@on developers with an ahrac@ve next-‐genera@on target (and significant challenges) – Aries interconnect with the Dragonfly network topology • Based on mature Cray XC30 architecture with Trinity introducing new architectural features – Intel Knights Landing (KNL) processors – Burst Buffer storage nodes – Advanced power management system sokware enhancements 9
- 10. Trinity PlaBorm • Trinity is enabling new architecture features in a produc@on compu@ng environment – Trinity’s architecture will introduce new challenges for code teams: transi@on from mul@-‐ core to many-‐core, high-‐speed on-‐chip memory subsystem, wider SIMD/vector units – Tightly coupled solid state storage serves as a “burst buffer” for checkpoint/restart file I/O & data analy@cs, enabling improved @me-‐to-‐ solu@on efficiencies – Advanced power management features enable measurement and control at the system, node, and component levels, allowing explora@on of applica@on performance/wah and reducing total cost of ownership • Managed Risk – Cray XC30 architecture minimizes system sokware risk and provides a mature high-‐speed interconnect – Haswell par@@on is low risk as technology; available in Fall CY14 – KNL is higher risk due to new technology, but provides a reasonable path, and resource, for code teams to transi@on to the many-‐core architecture 10
- 11. 11
- 12. Trinity Architecture Overview Metric Trinity Node Architecture KNL + Haswell Haswell Par@@on KNL Par@@on Memory Capacity 2.11 PB >1 PB >1 PB Memory BW >7PB/sec >1 PB/s >1PB/s +>4PB/s Peak FLOPS 42.2 PF 11.5 PF 30.7 PF Number of Nodes 19,000+ >9,500 >9500 Number of Cores >760,000 >190,000 >570000 Number of Cabs (incl I/O & BB) 112 PFS Capacity (usable) 82 PB usable PFS Bandwidth (sustained) 1.45 TB/s BB Capacity (usable) 3.7 PB BB Bandwidth (sustained) 3.3 TB/s 12
- 13. Compute Node specifica;ons Haswell Knights Landing Memory Capacity (DDR) 2x64=128 GB Comparable to Intel® Xeon® processor Memory Bandwidth (DDR) 136.5 GB/s Comparable to Intel® Xeon® processor # of sockets per node 2 N/A # of cores 2x16=32 60+ cores Core frequency (GHz) 2.3 N/A # of memory channels 2x4=8 N/A Memory Technology 2133 MHz DDR4 MCDRAM & DDR4 Threads per core 2 4 Vector units & width (per core) 1x256 AVX2 AVX-‐512 On-‐chip MCDRAM N/A Up to 16GB at launch, over 5x STREAM vs. DDR4 13
- 14. Trinity Capabili;es • Each par@@on will accommodate 1 to 2 large mission problems • Capability rela@ve to Cielo – 8x to 12x improvement in fidelity, physics and performance – > 30x increase in peak FLOPS – > 2x increase in node-‐level parallelism – > 6x increase in cores – > 20x increase in threads • Capability rela@ve to Sequoia – 2x increase in peak FLOPS – Similar complexity rela@ve to core and thread level parallelism
- 15. The Trinity Center of Excellence & Applica;on Transi;on Challenges • Center of Excellence – Work with select NW applica@on code teams to ensure KNL Par@@on is used effec@vely upon ini@al deployment – Nominally one applica@on per laboratory (SNL, LANL, LLNL) – Chosen such that they impact the NW program in FY17 – Facilitate the transi@on to next-‐genera@on ATS code migra@on issues – This is NOT a benchmarking effort • Intel Knights Landing processor – From mul@-‐core to many-‐core – > 10x increase in thread level parallelism – A reduc@on in per core throughput (1/4 to 1/3 the performance of a Xeon core) – MCDRAM: Fast but limited capacity (~5x the BW, ~1/5 the capacity of DDR4 memory) – Dual AVX-‐512 SIMD units: Does your code vectorize? • Burst Buffer – Data analy@cs use cases need to be developed and/or deployed into produc@on codes – Checkpoint/Restart should “just work”, although advanced features may require code changes
- 16. Trinity will be located at the Los Alamos Nicholas C. Metropolis Center for Modeling and Simula;on 16 Nicholas C. Metropolis Center for Modeling and Simula;on • Classified compu@ng • 15MW / 12MW water, 3MW air • 42” subfloor, 300 lbs/sqf • 80’x100’ (8,000 sqf) Trinity Power and Cooling • At least 80% of the plaborm will be water cooled • First large water cooled plaborm at Los Alamos • Concerns • Idle power efficiency • Rapid ramp up / ramp down load on power grid over 2MW
- 17. Trinity PlaBorm Schedule Highlights 2014-‐2016 17
- 18. Challenges and Opportuni;es • Applica@on transi@on work using next genera@on technologies (for Trinity, ATS-‐2, ATS-‐3, …) – Many-‐core, hierarchical memory, burst buffer • Opera@ng a large supercomputer using liquid cooling technology • Opera@ng Trinity using a mix of Haswell and KNL nodes • The Burst Buffer concepts and technology for improving applica@on efficiency and exploring other user cases • On the road to Exascale… • ….
- 19. Trinity System Summary • Trinity is the first instan@a@on of the ASC’s ATS plaborm • Trinity meets or exceeds the goals set by the ACES Design Team • Rela@ve to Cielo, Trinity will require applica@ons to transi@on to an MPI+X programming environment and requires increases in thread and vector level parallelism to be exposed • Trinity introduces Ac@ve Power Management, Burst Buffer storage accelera@on, and the concept of a Center of Excellence to ASC produc@on plaborms 19
- 20. Special thanks to the following for contribu@on of slides used in this talk • Doug Doerfler -‐ SNL • Thuc Hoang – NNSA ASC • And a cast of others……. 20
- 21. Ques;ons? • For more info visit the Trinity Web Site: trinity.lanl.gov 21