Idc Eu Study Slides 10.9.2010
Download as PPTX, PDF1 like564 views
The IDC recommendations report outlines a strategic agenda for the EU to enhance its leadership in high-performance computing (HPC) by 2020, aiming to boost scientific and economic competitiveness. Key recommendations include expanding HPC resources, ensuring better funding models, and enhancing collaboration between users and vendors to meet industry needs. It emphasizes the urgency of investing in HPC to avoid falling behind global competitors, especially as nations like China advance rapidly in supercomputing capabilities.
Idc Eu Study Slides 10.9.2010
- 1. IDC Recommendations Report: For EU HPC Leadership In 2020 October 2010 Authors: Earl Joseph, Steve Conway and Jie Wu Copyright 2010 IDC. Reproduction is forbidden unless authorized. All rights reserved.
- 2. Overall EU Study Strategic Goals Develop a complete strategic agenda to guide EU investments in HPC and to help drive European leadership in using and providing HPC Increase scientific and research competitiveness Increase the use of HPC for economic growth Bring together users and vendors to help ensure that offerings match users' requirements Enlarge/deepen the use of e-Infrastructure computing services to meet the growing demand from academia and industry Contract Number: 2009/S99-142914 EU Contract Manager: Bernhard FABIANEK, Performance Computing Officer European Commission , DG Information Society , Unit F03 — office BU25 04/087 25 Avenue Beaulieu, B-1160 Bruxelles © 2010 Oct-10
- 3. Overall Study Process Approach EU Additional Advice/ Executive IDC HPC Ideas Discussions Databases Extensive Survey Input Analysis #1: Interim Extensive From European Report: Analysis #2: Users, Vendors What is the and current April What are the Stakeholders situation and 2010 options, and European what should view points? be done for leadership? Technical Working Group Ideas, Suggestions, etc. Recommendations Report With SWOTS: Broader Inputs July 2010 From The Web Site © 2010 Oct-10 3
- 4. External Technical Committee Richard Blake, Acting Director - Computational Science and Engineering Department, Daresbury Science and Innovation Campus, UK Prof. Dr. Arndt Bode, Chairman of Board of Directors - Leibniz-Rechenzentrum, Germany Dr. Friedel Horst Willi Hossfeld , Professor Emeritus - Forschungszentrum Jülich, Germany Hervé Mouren, Managing Director - Teratec, France Christian Saguez, Teratec, France © 2010 Oct-10
- 5. One Reason Action Is Needed: A Concern: Will China Exceed The EU In Supercomputers? From: A June 30, 2010 interview with Jack Dongarra, by Sander Olson http://nextbigfuture.com/2010/06/jack-dongarra-interview-by-sander-olson.html Question: How quickly will China become a major player in the HPC space? Dongarra Answer: China is already a major player. In 2001, China had no machines in the top500 supercomputer list. They have more computing power than Japan, which is considered a supercomputing powerhouse, and “I predict that within a year they will have surpassed the entire European Union with its 27 nations.” China may soon have the fastest computer on the planet, and should soon have scores of systems in the top500. © 2010 Oct-10
- 6. #2 in China: Nebulae’s New Home – National Supercomputing Center in Shenzhen, China © 2010 Oct-10 6
- 7. Part 1: Interim Report: Field Research & Analysis Copyright 2010 IDC. Reproduction is forbidden unless authorized. All rights reserved.
- 8. Interim Report: Executive Overview • HPC use is indispensable for advancing both science and industrial competitiveness • Europe is under-investing in HPC, while other nations are growing their supercomputer investments dramatically • Even in 2009, the most difficult year of the global economic recession • The transition to petascale and exascale computing creates opportunities: • For Europe to return to the forefront of development for the next generation of research and HPC software technologies © 2010 Oct-10 8
- 9. HPC Worldwide Market Sizes: Supercomputers (Over $500K) Europe declined from 34.4% in 2007, to 24.8% in 2009 T ABLE 6 Worldwide HPC Supercomputer System Revenue ( €000) by Region, 2005 –2009 2005 2006 2007 2008 2009 CAGR (05–09) Total WW Revenue 2,160,829 1,925,165 2,011,793 2,014,596 2,527,058 4.0% North America Revenue 1,043,865 903,948 932,183 1,031,201 1,291,493 5.5% Europe Revenue 614,307 582,989 692,038 592,535 627,732 0.5% ** Percent of WW 28.4% 30.3% 34.4% 29.4% 24.8% Asia/Pac Revenue 249,244 204,639 228,972 219,970 226,608 -2.4% Japan Revenue 231,745 206,965 122,733 137,872 348,448 10.7% Rest of World Revenue 2,669 8,594 14,464 14,692 13,362 49.6% Source: IDC, 2010 © 2010 Oct-10 9
- 10. HPC Market Sizes: Supercomputers By Country FIGURE 2 EU Total Over $3M/€2.25M + High -End Supercomputer Revenue Mix by Country, 2009 All Others (9.2%) Spain (5.0%) Germany (31.5%) Italy (15.2%) France (19.8%) U.K. (16.3%) Sweden (2.9%) Source: IDC, 2010 © 2010 Oct-10 10
- 11. Interim Report: EU Purchases By Sector In 2009, EU spending was greatest for systems in the following application sectors, in order of the amount spent: University/academic ($370 million) Bio-sciences ($354 million) CAE ($337 million) Government labs ($302 million) Defense ($208 million) © 2010 Oct-10 11
- 12. Interim Report: EU Purchases By Sector TABLE 28 EU Countries HPC Revenue (€000) by Application, 2005 –2009 Application 2005 2006 2007 2008 2009 CAGR (05–09) Bio-Sciences 302,182 331,635 348,304 322,705 265,509 -3.2% CAE 261,610 294,130 316,270 299,927 252,487 -0.9% Chemical Engineering 43,681 51,332 57,368 56,064 48,493 2.6% DCC & Distribution 97,497 102,734 102,900 89,934 69,348 -8.2% Economics/Financial 42,898 50,917 56,969 55,636 48,018 2.9% EDA 126,954 141,789 150,907 140,428 115,781 -2.3% Geosciences and Geo-engineering 107,260 119,138 126,176 117,248 96,894 -2.5% Mechanical Design and Drafting 38,718 41,583 42,306 37,792 29,774 -6.4% Defense 131,623 155,652 175,132 174,738 155,639 4.3% Government Lab 309,590 325,110 324,815 286,797 226,579 -7.5% Software Engineering 4,472 4,474 4,240 3,520 2,563 -13.0% Technical Management 22,192 22,058 20,479 16,417 11,285 -15.6% University/Academic 339,481 365,139 375,398 341,381 278,036 -4.9% Weather 87,922 93,970 96,225 87,239 70,561 -5.4% Other 648 2,484 4,422 5,768 6,138 75.4% Total Revenue 1,916,730 2,102,146 2,201,912 2,035,594 1,677,105 -3.3% Source: IDC, 2010 © 2010 Oct-10 12
- 13. Study Highlights: Successful HPC Programs (As Seen By HPC Experts) Most successful international HPC programs: 1. U.S. Department of Energy — SciDAC (10) 2. U.S. Department of Energy — INCITE (9) 3. U.S. National Science Foundation – NSF (8) 4. Japan's RIKEN /Keisoku Project (7) 5. U.S. Department of Defense — DARPA (5) 6. U.S. Department of Defense — HPC Modernization Program (3) 7. PRACE (2) 8. IESP (2) Most successful EU HPC Programs: 1. PRACE (14) 2. DEISA (10) 3. CEA-DAM (4) 4. CERN (4) 5. HPC Europa (4) 6. Blue Brain –EPFL (2) 7. EGEE (2) 8. ESA (2) 9. Forschungszentrum Juelich (2) 10. GEANT (2) © 2010 Oct-10 13
- 14. Study Highlights: Expertise Most Needed The survey respondents see the areas of expertise most needed from HPC user organizations as: Expertise in parallel programming for highly parallel HPC systems Expertise in creating advanced software algorithms The ability to port and optimize applications for new hardware architectures, including heterogeneous architectures that include newer processor types © 2010 Oct-10 14
- 15. Study Highlights: Funding Models Currently Used TABLE 58 What is Your Company/Organization's Current Funding Approa ch for HPC? Respondents Percentage of Sample Our internal funding is helped by government funding from within our country 34 65.4% Our HPC is primarily funded by our own organization 26 50.0% We receive multi-country funding, but it is less than 20% of our HPC costs 5 9.6% We have public-private partnership(s) that help to fund our HPC 5 9.6% We receive multi-country funding, and it is more than 20% of our HPC costs 3 5.8% Note: Multiple responses were allowed. 52 sites responded to this question. Source: IDC, 2010 © 2010 Oct-10 15
- 16. Study Highlights: Desired Funding Models The survey respondents' ideas for the best funding models to pursue HPC goals in Europe included: Provide an EU-wide framework (such as PRACE) to drive toward HPC goals Create sustained, multi-year funding Focus funding most heavily on a limited number of well-defined scientific and industrial problems, and use a cost-benefit analysis to identify these problem domains Focus more on software than hardware Base access heavily on grants awarded through peer-reviewed proposals © 2010 Oct-10 16
- 17. Study Highlights: Poor Funding Models Funding Models That Have Failed in HPC: Models based on short-term goals Funding heavily focused on many small projects rather than a few large ones Funding procurements that include protectionist measures Models that impose overly burdensome bureaucratic requirements to gain access to HPC resources. Models that require organizations to pay to use HPC resources Models that require countries to "buy their way in" Models aimed at developing new HPC system architectures © 2010 Oct-10 17
- 18. From The Interim Report: Top Areas That Stakeholders Said The EU Should Focus On What are the most important approaches for EU HPC leadership? Percentage Number of Of Approach Responses Responses Making world-class HPC resources more widely available to the EU scientific and engineering communities 57 93.4% Advancing scientific leadership by using HPC to solve some of the world's most challenging problems 50 82.0% Making HPC more readily available for the first time to small and medium-size businesses (including industrial supply chains, small educational sites, etc.) 33 54.1% Having many very large supercomputers, e.g., being at or near the top of the Top500 list of the world’s most powerful supercomputers (www.top500.org) 25 41.0% Building an EU-based HPC vendor community with world- class capabilities in important areas (hardware, software, storage, networking, etc.) 24 39.3% © 2010 Oct-10 18
- 19. Part 2: Final Report: Strategic Agenda Recommendations Copyright 2010 IDC. Reproduction is forbidden unless authorized. All rights reserved.
- 20. Proposed Vision For EU HPC Leadership Note: Over 200 HPC experts across the EU have provided ideas, insights and suggestions in the creation of this report and proposal. © 2010 Oct-10
- 21. The Proposed Primary Vision For EU HPC Leadership By 2020 and Beyond Provide World-Class HPC Resources To Make EU Scientists, Engineers And Analysts The Most Productive and Innovative In The World In Applying HPC To Advance Their Research In The Pursuit Of Scientific Advancement And Economic Growth © 2010 Oct-10 21
- 22. Potential Results From The Vision By the year 2020, the EU HPC strategy has enabled the following progress to occur: 1. Europe is recognized as a hotbed for new science and engineering research. 2. Europe's leadership in the targeted areas has created many new jobs in science and industry, and has caused the national economies to grow faster. • Europe's move to the forefront of progress in other areas has preserved many existing jobs in both science and industry. 3. Europe is the world leader in important high-end HPC technologies. • Europe leads the world in scalable algorithms and software applications, and in tools to make HPC systems easy-to-use and to make researchers highly innovative and productive. © 2010 Oct-10 22
- 23. Vision Results Vision Results Provide world-class HPC • Adds billions of euros a year to resources European economies by 2020 To make EU scientists, • Make the EU a hotbed for scientific and engineers and analysts engineering research The most productive and • Invigorate EU institutions by making EU innovative in the world researchers the most productive in the In applying HPC to advance world their research • Expand the HPC ecosystem across the In pursuit of scientific EU creating a vibrant HPC supplier advancement and economic economy growth © 2010 Oct-10
- 24. Actions Required © 2010 Oct-10
- 25. Europe Needs To Invest In/Support A Robust HPC Industry With Hardware, Software, Etc. Hardware: • Develop unique capabilities for use in standards-based systems, e.g. interconnects, system design, packaging, power and cooling, alternative processors, SSDs, etc. • IDC recommends that the EU avoids making any excessively large R&D or NRE investments (in the €1+ billion range) Software: • Exploit world-class strengths in Europe • Algorithms, compliers, system software, file systems, applications, libraries, etc. Storage: • Advanced storage systems and file systems Networking: • Internal and external to the exascale system, high-bandwidth, low latency communications, grids and perhaps clouds © 2010 Oct-10 25
- 26. Overall IDC Recommendations: Actions Required To Achieve The Vision 1. First is the need for expanding the number and size of HPC resources across the EU (including broader access to the tools by all EU researchers). • An expanded PRACE can address this requirement. 2. Second is to provide broader access to industrial HPC users. 3. Third is to make HPC users more productive by creating the world's best tools, training and development environment. • Requires a new initiative (HPC development labs/test-beds). 4. Fourth is to attract more students into scientific, engineering and HPC fields and to attract more experts around the world to join in EU projects. • Requires additional funding and a "Magnet" program. 5. Fifth is the need to increase funding in developing next generation Exascale software. 6. Sixth is to target a few strategic application areas for global leadership. © 2010 Oct-10 26
- 27. Actions Required To Achieve The Vision: Enhancing And Expanding PRACE Recommended Enhancements to PRACE: 1. First is increasing funding to provide for more very-large systems and for more systems across the EU. 2. Second is increasing access to researchers in science and industry. • And to include more countries. 3. Third is the enhanced charter and perhaps a new name: • Expand the mission, stature and funding of the PRACE program to make it the central EU organization responsible for pursuing the mainstream system acquisition and operating EU HPC strategy on behalf of, and in collaboration with, the member states. • The creation of an EU HPC system strategy elevates HPC to an appropriate, larger role within Europe. • An expanded central organization is needed to pursue the strategy and to signal the EU's increased commitment to HPC leadership. © 2010 Oct-10 27
- 28. Actions Required To Achieve The Vision: HPC Development Labs/Test-bed Centers These new HPC development labs/test bed centers are needed to address these weaknesses in HPC today: • Most large HPC systems are very hard to use buy all but the most talented experts. • The productivity in using highly parallel systems is very low, no matter which metric ones uses: • Time to get an application up and running vs. the usefulness of the results. • Time to optimize codes vs. the speed-up obtained. • Percentage of the system that can be effectively used by a user's job. • There are a limited number of scientists, engineers and analysts in the EU. • While other countries like China and India have strong growth in graduates, making the productivity of these experts a critical deciding factor in the advancement of science and research. © 2010 Oct-10 28
- 29. Actions Required To Achieve The Vision: HPC Development Labs/Test-bed Centers What the new HPC development labs/test-bed centers could look like: • These would be dedicated HPC centers with the goals of helping users, researchers and companies make their codes and products highly scalable and highly productive. • These HPC systems would only be used in a test and development mode. • They may crash hourly as new algorithms, new codes or even new hardware is being tested, optimized and made resilient. • The systems need to be large, but not extremely large. • The HPC development test-beds should have major training and teaching responsibilities. They should offer classes in advance HPC, again with a focus on making users and products more productive. • This requires a strong technical staff, along with teachers and HPC experts across multiple domains. © 2010 Oct-10 29
- 30. Actions Required To Achieve The Vision: HPC Development Labs/Test-bed Centers What the new HPC development labs/test-bed centers could look like: • To support individual users and researchers across the EU. • For example, if a scientist has an idea on how to make their codes run at a larger scale, they could make use of the test-bed systems and staff to figure out new algorithms, test them at scale, make improvements and then take the results back to their home organization. • They would also support the broader EU vendor community and ecosystem. • For example if a vendor like Bull decides to research a new interconnect for exascale systems, they could use these HPC development test-beds for testing out the new interconnects by physically installing the new hardware, and the running tests, de- bugging and making their product more resilient. • The EU should help support the broader HPC ecosystem by making these resources free to all, for development and testing work only. © 2010 Oct-10 30
- 31. Actions Required To Achieve The Vision: A new ―Magnet‖ Program To accomplish the major objective on making Europe the hotbed for scientific research, the EU needs to fund a number of "Magnet" programs to attract top scientists, recent graduates, and other researchers to European institutions. The magnet program should include: • Funding of highly prestigious positions/seats at top EU educational institutions. These need to include sizeable research grants to attract the top talent. • Create a new degree-like position between a Ph.D. and a professor. And invite selected top researchers and top new graduates to attend these degree programs at no cost. • Broad incentives should be used including prestigious positions and titles, open access to leading scientific projects, etc. • These positions need an exciting title and full press coverage. The goal should be to start ASAP with 10's of these new positions, growing quickly to 100's per year. © 2010 Oct-10 31
- 32. Actions Required To Achieve The Vision: Purchase vs. Develop Exascale Systems • IDC recommends that the EU buys the 4th or 5th exascale system, and does not invest in developing the first 1, 2 or 3 exascale systems in the world. • This strategy could save a billion plus euros • –> that can be used in buying more systems and making EU researchers more productive. • Europe could save even more by instead aiming to purchase a near-exascale system in this timeframe, but 1 year later. • We estimate that this would substantially reduce the technical challenges (e.g., hardware/software scaling) and associated costs. • Such that an investment on the order of €150 to €200 million per system would likely suffice. • The resultant near-exascale supercomputer would still sustain unrivaled performance in the targeted application domain, and still attract the best researchers and collaborations. © 2010 Oct-10 32
- 33. Investments Required © 2010 Oct-10
- 34. Investments Required It is recommended that the EU and the nations make HPC a higher priority and step up to either the ―Full leadership level" or at least the ―Funding to reach major goals level" scenario level. • This would require a net new investment reaching 600 million euros a year within five years. © 2010 Oct-10 34
- 35. Investments Required And Scenarios TABLE 4 Alternative HPC Funding Scenarios: Yearly Funding ADDITIONS By Year Five (Millions of euros added per year) Full Leadership Funding To Reach Partial Minimal Increase Funding Level Major Goals Level Funding Level Funding Level HPC System Funding Increases 250 210 120 50 HPC Development Test-beds (H/W) 50 40 25 0 HPC Development Test-beds (People) 75 60 35 0 Exascale Software Development 150 125 55 25 Scientific Talent Magnet Program 75 60 25 0 Total Yearly Funding Increase 600 495 260 75 Note: These figures include HPC funding paid by the EU, by member nations and contributions by vendors. Source: IDC, 2010 © 2010 Oct-10 35
- 36. In Summary: Actions Vision Results Actions Vision Results • Expand the number, size • Add billions of euros a and access to HPC year to European resources across the EU Provide world-class HPC resources economies by 2020 • Add a focus on industrial • Make the EU a hotbed users To make EU scientists, engineers and analysts for scientific and • Create a set of HPC engineering research development labs/test-bed The most productive and innovative in the • Invigorate EU institutions centers by making EU world • Attract more students and researchers the most experts from around the In applying HPC to productive in the world world to join in EU projects advance their research • Expand the HPC • Fund development of next In pursuit of scientific ecosystem across the generation Exascale S/W advancement and EU creating a vibrant economic growth HPC supplier economy • Target strategic application areas for global leadership © 2010 Oct-10 36
- 37. The Project’s Web Site: www.hpcuserforum.com/EU This now has the interim report and will soon have the final report –> and will ask for feedback © 2010 Oct-10 37
- 38. Acknowledgments We greatly thank the following for all of the great ideas, feedback, suggestions and advice: • EC (Kostas Glinos, Bernhard Fabianek, et al.) • The Technical and Strategy Committee • All of the members of broader EU HPC Community that contributed to this study Questions? © 2010 Oct-10 38
- 39. Questions? Please email: hpc@idc.com Or check out: www.hpcuserforum.com www.idc.com © 2010 Oct-10 39