HPC in the cloud comes of age - Red Oak HPC Seminar
Download as PPTX, PDF1 like378 views
The document presents insights from a seminar on the evolution and current state of High-Performance Computing (HPC) in the cloud, highlighting historical challenges and advancements in cloud HPC technology. It discusses findings from the 2017 UK national e-infrastructure survey and outlines future trends, including the rise of hyperscale cloud providers and the increasing importance of container technologies. The seminar also emphasizes Jisc's role in supporting higher education and research institutions in navigating these developments.
HPC in the cloud comes of age - Red Oak HPC Seminar
- 1. HPC in the cloud comes of age Martin Hamilton 1HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017
- 2. HPC in the cloud comes of age 1. The journey to cloud HPC 2. UK national e-Infrastructure survey › 2017 survey results sneak preview 3. What’s next? 13/09/2017 HPC in the cloud comes of age - Red Oak Seminar - September 2017 2
- 3. About Jisc » Not-for-profit company serving the HE, FE and skills community: › 18 million users of shared services like Janet, eduroam, JiscMail, shared data centres › Devising national deals with IT vendors and publishers, e.g. routers and switches framework, AWS › Providing advice & guidance to the sector on digital technologies in research and education » My role: › Leading a small group exploring emerging technologies like Augmented Reality and brain computer interfaces › Helping institutions devise/implement digital strategies and building evidence base around new technologies › Working with e-Infrastructure community via Project Directors Group (formerly ran HPC Midlands for EPSRC) 3HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017
- 4. » Not-for-profit company serving the HE, FE and skills community: › 18 million users of shared services like Janet, eduroam, JiscMail, shared data centres › Devising national deals with IT vendors and publishers, e.g. routers and switches framework › Providing advice & guidance to the sector on digital technologies in research and education » My role: › Leading a small group exploring emerging technologies like Augmented Reality and brain computer interfaces › Helping institutions devise/implement digital strategies and building evidence base around new technologies › Working with e-Infrastructure community via Project Directors Group (formerly ran HPC Midlands for EPSRC) About Jisc 4HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017
- 5. » Not-for-profit company serving the HE, FE and skills community: › 18 million users of shared services like Janet, eduroam, JiscMail, shared data centres › Devising national deals with IT vendors and publishers, e.g. routers and switches framework › Providing advice & guidance to the sector on digital technologies in research and education » My role: › Leading a small group exploring emerging technologies like Augmented Reality and brain computer interfaces › Helping institutions devise/implement digital strategies and building evidence base around new technologies › Working with e-Infrastructure community via Project Directors Group (formerly ran HPC Midlands for EPSRC) About Jisc 5HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017
- 6. 1.The journey to cloud HPC 13/09/2017 HPC in the cloud comes of age - Red Oak Seminar - September 2017 6
- 7. The journey to cloud HPC How we got here, part 1: » Historically software was typically written using high performance languages like Fortran and C » …but workflows would often combine several of these tools using scripting languages like Python » The results were often quite brittle and difficult to transport from one machine to another - e.g. version dependencies » Meanwhile virtualisation techniques had been making a resurgence in enterprise IT and quickly became standard operating procedure for devops » Computer Science community initiatives like Research Object spring up… 7HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 http://www.researchobject.org
- 8. The journey to cloud HPC How we got here, part 2: » The virtual machine image became the common unit of currency for enterprise private clouds (e.g.VMWare) and public clouds (Azure, AWS, etc) » VM catalogues available from public cloud providers like AWS Marketplace + community initiatives like Bio-Linux » Growing popularity of orchestration tools likeAnsible,Vagrant and Puppet » But HPC sysadmins reluctant to provide VM environment on their machines › Performance impact? For everyone? 8HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 http://environmentalomics.org/
- 9. The journey to cloud HPC How we got here, part 3: » But not all jobs are pedal-to-the-metal, requiring maximum machine performance » Lots of HPC (and HTC, and…) users simply want to run a particular app, and are ecstatic if it’s faster than on their laptop » HPC providers initially dubious that a cloud environment could be superimposed, but then… › Growing popularity of OpenStack as an underpinning in-house platform › Creation ofVM oriented private clouds › Growing acceptability of public clouds 9HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 http://www.jasmin.ac.uk/
- 10. 2. UK national e-Infrastructure survey 13/09/2017 HPC in the cloud comes of age - Red Oak Seminar - September 2017 10
- 11. UK national e-Infrastructure survey 11HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 2017 survey report being developed Previous survey reports available from www.hpc-sig.org
- 12. UK national e-Infrastructure survey 12HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 Aberystwyth University Cardiff University DiRAC HPC Facility Diamond Light Source Ltd Durham University EMBL-European Bioinformatics Institute Edinburgh Parallel Computing Centre (EPCC) eMedLab Francis Crick Institute GridPP HPC Midlands+ Hartree Centre Institute of Cancer Research King's College London Lancaster University Loughborough University MRC Laboratory of Molecular Biology N8HPC Newcastle University Norwich BioScience Institutes (NBI) Queens University of Belfast Science and Technology Facilities Council (STFC) Supercomputing Wales Swansea University University College London University of Bath University of Birmingham University of Bristol University of Cambridge University of Dundee University of Edinburgh University of Glasgow University of Hull University of Leeds University of Leicester University of Liverpool University of Manchester University of Nottingham University of Oxford University of Portsmouth University of Sheffield University of Southampton University of St Andrews University of Strathclyde University of Sussex University of Warwick University of York Wellcome Trust Centre for Human Genetics Wellcome Trust Sanger Institute 2017 survey respondents
- 13. UK national e-Infrastructure survey 13HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 Word cloud of NeI service descriptions
- 14. UK national e-Infrastructure survey 14HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 Relationship with REF Units of Assessment
- 15. UK national e-Infrastructure survey 15HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017
- 16. 16HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 UK national e-Infrastructure survey
- 17. 17HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 UK national e-Infrastructure survey
- 18. UK national e-Infrastructure survey 18HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017
- 19. UK national e-Infrastructure survey 19HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017
- 20. UK national e-Infrastructure survey 20HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017
- 21. 3. What’s next? 13/09/2017 HPC in the cloud comes of age - Red Oak Seminar - September 2017 21
- 22. What’s next? » Hyperscale cloud providers offering: › Performant interconnects, e.g.Azure RDMA › Latest hardware, e.g. Azure FPGAs + “very fat nodes” › Azure ExpressRoute + AWS Direct Connect peerings (Jisc facilitating) » Some consolidation of private clouds into domain / community facilities, e.g. CLIMB › Authorisation / access challenges (JiscAssent + Safe Share) » Containers gradually take over fromVM images - cf. Open Containers Initiative › Do we need a clearinghouse, or willGitHub do? (for open source) 22HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017 https://www.opencontainers.org
- 23. Continue the conversation… 23 Except where otherwise noted, this work is licensed under CC-BY Martin Hamilton Futurist, Jisc @martin_hamilton martin.hamilton@jisc.ac.uk HPC in the cloud comes of age - Red Oak Seminar - September 201713/09/2017