HPC Facility Designing for next generation HPC systems Ram Nagappan Intel Final
- 1. HPCFacility:Designingfornext generationHPCsystemsRamkumar Nagappan System Architect Technical Computing Group System Architecture and Pathfinding Intel May 11, 2015 6th European workshop on HPC centre infrastructures, Stockholm, Sweden
- 2. INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. This document contains information on products in the design phase of development. All products, computer systems, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel product plans in this presentation do not constitute Intel plan of record product roadmaps. Please contact your Intel representative to obtain Intel’s current plan of record product roadmaps. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as STREAM , NPB, NAMD and Linpack, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Intel does not control or audit the design or implementation of third party benchmarks or Web sites referenced in this document. Intel encourages all of its customers to visit the referenced Web sites or others where similar performance benchmarks are reported and confirm whether the referenced benchmarks are accurate and reflect performance of systems available for purchase. Relative performance is calculated by assigning a baseline value of 1.0 to one benchmark result, and then dividing the actual benchmark result for the baseline platform into each of the specific benchmark results of each of the other platforms, and assigning them a relative performance number that correlates with the performance improvements reported. Intel, Xeon and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others Copyright © 2014 Intel Corporation. All rights reserved. Notices 2
- 3. OptimizationNoticeOptimization Notice Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 * Other names and brands may be claimed as the property of others. Copyright © 2014 Intel Corporation. All rights reserved.
- 4. Agenda • Exponential Growth in Performance • Trends • Cores & Threads • Power Efficiency • Process Improvement • Integration to increase system performance • I must need a huge datacenter –right? • Quick Survey • How many Square Meter/PFlop? • Past, Present & Future • Power Delivery Challenges in the Horizon • Technologies to measure component/node power • Summary 4
- 5. ExponentialGrowthinPerformance Advancement in key areas to reach Exascale: • Microprocessors • Fabrics • Memory • Software • Power Management • Reliability 5 Source:Top500.org
- 7. 7 Source: SICS Multicore Day’ 14
- 8. 8 Source: SICS Multicore Day’ 14
- 9. 9
- 10. More Integration 3+ TFLOPS1 In One Package Parallel Performance & Density On-Package Memory: up to 16GB at launch 5X Bandwidth vs DDR47 Compute: Energy-efficient IA cores2 Microarchitecture enhanced for HPC3 3X Single Thread Performance vs Knights Corner4 Intel Xeon Processor Binary Compatible5 1/3X the Space6 5X Power Efficiency6 . . . . . . Integrated Fabric Intel® Silvermont Arch. Enhanced for HPC Processor Package Source:SC’14 … Knights Landing (Next Generation Intel® Xeon Phi™ Products) Jointly Developed with Micron Technology System level benefits in cost, power, density, scalability & performance
- 11. HigherPerformance&Density A formula for more performance…. advancements in CPU architecture advancements in process technology integrated in-package memory integrated fabrics with higher speeds switch and CPU packaging under one roof all tied together with silicon photonics = much higher performance & density 11
- 12. Agenda • Exponential Growth in Performance • Trends • Cores & Threads • Power Efficiency • Process Improvement • Integration to increase system performance • I must need a huge datacenter –right? • Quick Survey • How many Square Meter/PFlop? • Past, Present & Future • Power Delivery Challenges in the Horizon • Technologies to measure component/node power • Summary 12
- 14. QuickSurvey In what year, do you expect to see “rack” level performance exceed 1 PF? a) 2016 b) 2018 c) 2020 d) 2022 14 Note: 2018 is not an official Intel projection
- 15. Designingforthefuture What does “Large” mean? A great example of reduced footprint with an equal performance: 15 Credit: Thanks to LRZ - Helmut Satzger for the video https://www.youtube.com/watch?v=qirUUlXR6XQ
- 16. 16
- 17. How many Sq mt Per PFlop? System Feature LRZ Phase 1 Details Year 2012 Performance Pflop/s 3.2 # of Nodes 9216 Power 2.3 MW Facility area for Compute Clusters ~546 sq mt How many Sq mt Per PFlop 171 LRZ SuperMUC System Source: lrz.de
- 18. How many Sq mt Per PFlop? System Feature LRZ Phase 1 Details LRZ Phase 2 Details Year 2012 2015 Performance Pflop/s 3.2 3.2 # of Nodes 9216 3096 Power 2.3 MW 1.1 MW Facility area for Compute Clusters ~546 sq mt ~182 sq mt (*Estimated) How many Sq mt Per PFlop 171 57 LRZ SuperMUC System Source: lrz.de Phase 2 Phase 1
- 19. How many Sq mt Per PFlop? System Feature LRZ Phase 1 Details LRZ Phase 2 Details Trinity Details Year 2012 2015 FY 2015/16 Performance Pflop/S 3.2 3.2 42.2 # of Nodes 9216 3096 >19,000 (Haswell, KNL) Power 2.3 MW 1.1 MW < 10 MW Facility area for Compute Clusters 546 sq mt ~182 sq mt (*Estimated) < 483 sq mt How many Sq mt Per PFlop 171 57 11.4 LRZ SuperMUC System Source: lrz.de trinity.lanl.gov Phase 2 Phase 1
- 20. How many Sq mt Per PFlop? System Feature LRZ Phase 1 Details LRZ Phase 2 Details Trinity Details Aurora Details Year 2012 2015 FY 2015/16 2018 Performance Pflop/S 3.2 3.2 42.2 180 # of Nodes 9216 3096 (Haswell) >19,000 (Haswell, KNL) >50,000 KNH Power 2.3 MW 1.1 MW < 10 MW 13 MW Facility area for Compute Clusters ~546 sq mt ~182 sq mt (*Estimated) < 483 sq mt ~279 sq mt How many Sq mt Per PFlop 171 57 11.4 1.5 Source: lrz.de LRZ SuperMUC System Phase 2 Phase 1 Past 2015/16 Future Note: not an official Intel projection
- 21. Significant improvement in Power efficiency System Feature LRZ Phase 1 Details LRZ Phase 2 Details Trinity Details Aurora Details Year 2012 2015 FY 2015/16 2018 Performan ce -Pflop/s 3.2 3.2 42.2 180 # of Nodes 9216 3096 >19,000 (Haswell, KNL) >50,000 Power 2.3 MW 1.1 MW < 10 MW 13 MW System Power Efficiency (GFlops/W att) 1.4 2.9 4.2 13.8 LRZ SuperMUC System Past 2015/16 Future Source: lrz.de Phase 2 Phase 1 Note: not an official Intel projection
- 22. 33 PF (#1 now) System in 2018 timeframe •What is the floor space and power required to build today’s No.1 System in Top 500 list in 2018?
- 23. 33 PF (#1 now) System in 2018 timeframe • What is the floor space and power required to build today’s No.1 System in Top 500 list in 2018 • Based on earlier projections/estimations • Very High Density • Facility area for Compute Clusters is ~50 Sq mt • Power – ~2.4 MW Very High Density
- 24. 33 PF (#1 now) System in 2018 timeframe • What is the floor space and power required to build today’s No.1 System in Top 500 list in 2018 • Based on earlier projections/estimations • Very High Density • Facility area for Compute Clusters is ~50 Sq mt • Power – ~2.4 MW • If we assume lower rack density • 50% lower rack density – High Density • Facility area for Compute Clusters is 100 Sq mt Very High Density High Density
- 25. 33 PF (#1 now) System in 2018 timeframe • What is the floor space and power required to build today’s No.1 System in Top 500 list in 2018 • Based on earlier projections/estimations • Very High Density • Facility area for Compute Clusters is ~50 Sq mt • Power – ~2.4 MW • If we assume lower rack density • 50% lower rack density – High Density • Facility area for Compute Clusters is 100 Sq mt • 75% lower rack density – High to Medium Density • Facility area for Compute Clusters is 200 Sq mt Do we still need a large data center? High Density Very High Density Medium Density
- 26. I must need a huge data center – Right? 26 • Facility area for Compute Cluster does not have to be huge. • Don’t forget • If Storage is going to be large then you will need additional floor space. • If you are going to be using Xeon instead of Xeon Phi then you may need additional floor space.
- 27. HighDensityDatacenter-Challenges • Performance density • PF / rack – smaller HPC floor space • Weight density • 500 lbs/sf ~ 2500 kg/m2 – Likely needed • Power density • >100 kW / rack fed with 480Vac or 400Vac feeds • Pipe size • ~40 mm to/from each rack • Ratios – Raised Floor vs Infrastructure Space • m2/m2 < 1 27
- 28. Agenda • Exponential Growth in Performance • Trends • Cores & Threads • Power Efficiency • Process Improvement • Integration to increase system performance • I must need a huge datacenter –right? • Quick Survey • How many Square Meter/PFlop? • Past, Present & Future • Power Delivery Challenges in the Horizon • Technologies to measure component/node power • Summary 28
- 29. PowerDeliveryChallengesinthehorizon Maximum Power Cap • Ex: 10 MW • Several reasons • Peak Shedding • Reduction in renewable energy Power rate of change • Ex: Hourly or Fifteen minute average in platform power should not exceed by X MW. Controlled Power Ramp up/down Power Monitoring at Component, Node, Rack/Cabinet, System 29
- 31. IntelXeon/PhiComponentLevelEnergyMeasurement 31 RAPL (Running Average Power Limit) to Monitor and Limit – CPU and Memory Power
- 32. Intel®NodeManager Power Telemetry • Total platform power • Individual CPU, Memory and Xeon Phi power domains Thermal Telemetry • Inlet & Outlet Airflow temperature • Volumetric Airflow Utilization Telemetry • Aggregate Compute Utilization per sec • CPU, Memory and I/O Utilization Metrics Power Controls • Power limit during operation • Power limit during boot BMC Intel® Node Manager Firmware embedded in PCH’s “Manageability Engine” Console (e.g. Intel DCM) Xeon Xeon PCH (ME)
- 33. SUMMARY • Trends • Continuous growth in Cores & Threads • Continuous & significant improvement in Power Efficiency • More Integration to improve performance and efficiency • Designing for the future • It’s going to be Large! kg/rack, kW/rack, perf/rack, power ramps and peak, pipe sizes, m2/m2 • It may get Smaller! Cluster footprint • Power Challenges in the horizon • Power Limiting, Power rate of change • Technologies to measure component and Node power 33
- 34. 34
- 35. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm Intel, Intel Xeon, Intel Xeon Phi™, Intel® Atom™ are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. Copyright © 2014, Intel Corporation *Other brands and names may be claimed as the property of others. Intel does not control or audit the design or implementation of third party benchmark data or Web sites referenced in this document. Intel encourages all of its customers to visit the referenced Web sites or others where similar performance benchmark data are reported and confirm whether the referenced benchmark data are accurate and reflect performance of systems available for purchase.The cost reduction scenarios described in this document are intended to enable you to get a better understanding of how the purchase of a given Intel product, combined with a number of situation-specific variables, might affect your future cost and savings. Nothing in this document should be interpreted as either a promise of or contract for a given level of costs. Intel® Advanced Vector Extensions (Intel® AVX)* are designed to achieve higher throughput to certain integer and floating point operations. Due to varying processor power characteristics, utilizing AVX instructions may cause a) some parts to operate at less than the rated frequency and b) some parts with Intel® Turbo Boost Technology 2.0 to not achieve any or maximum turbo frequencies. Performance varies depending on hardware, software, and system configuration and you should consult your system manufacturer for more information. *Intel® Advanced Vector Extensions refers to Intel® AVX, Intel® AVX2 or Intel® AVX-512. For more information on Intel® Turbo Boost Technology 2.0, visit http://www.intel.com/go/turbo All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. Optimization Notice Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2®, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 Legal Disclaimer * Other names and brands may be claimed as the property of others. Copyright © 2014 Intel Corporation. All rights reserved. 35