OpenNebulaConf2017EU: Transforming an Old Supercomputer into a Cloud Platform by Alvaro Simon, UGent
1 like196 views
The document discusses the transformation of an old Flemish tier-1 supercomputer into a public cloud platform using OpenNebula to meet specific user requirements and reuse existing hardware. Key features include virtualization capability, enhanced network configuration, secure user access, and storage management via Ceph. Lessons learned highlight the effectiveness of the network setup and the quick deployment of pilot services for biomedical research and other user needs.
OpenNebulaConf2017EU: Transforming an Old Supercomputer into a Cloud Platform by Alvaro Simon, UGent
- 2. Transforming an Old SupercomputerTransforming an Old Supercomputer into a Cloud Platforminto a Cloud Platform ÁLVARO SIMÓN GARCÍA - HPC UGENT OpenNebula Conference – October 24th 2017 Madrid
- 3. CONTENTS ‒ Why? ‒ From HPC to a Cloud Cluster ‒ Pilot Users ‒ Lessons learned
- 4. WHY? OpenNebula Conference – October 24th 2017 Madrid 4
- 5. Why? – Some users have specific requirements that are not offered by the HPC infrastructure and services. – Reuse of hardware that is no longer used in production clusters, but is still sufficiently capable of running a cloud environment. – Provide a public Cloud Infrastructure to our users in a short period of time. OpenNebula Conference – October 24th 2017 Madrid 5
- 6. Why? Old Flemish Tier-1 Supercomputer (Muk)*: ● 528 nodes with two Xeon E5-2670 processors (2.6GHz, 8 cores per processor). ● 64GiB of memory per node. ● FDR Infiniband. ● 400TB storage. OpenNebula Conference – October 24th 2017 Madrid 6 *Muk: We always assign Pokémon names to all our clusters
- 7. FROM HPC TO CLOUD OpenNebula Conference – October 24th 2017 Madrid 7
- 8. OpenNebula Conference – October 24th 2017 Madrid 8
- 9. OpenNebula Conference – October 24th 2017 Madrid 9
- 10. OpenNebula Conference – October 24th 2017 Madrid 10
- 11. From HPC to Cloud ● A few things to take into account: – Use As Is. No hardware upgrades or changes. We use what we have. – Security. Isolate VMs network traffic, provide a SSO, synchronise HPC users with the Cloud infrastructure. – Time. We are restricted timewise. – End of Warranty. The Hardware is no longer supported by the vendor. OpenNebula Conference – October 24th 2017 Madrid 11
- 12. Use As Is (a regular Muk Tier-1 worker node): OpenNebula Conference – October 24th 2017 Madrid 12 InfiniBand Switch Ethernet Switch Eth0 (10.131.x.x)IB0 (10.132.x.x) Low latency network used by MPI jobs Gigabit Ethernet used by management, monitoring and the job scheduler. HP ProLiant SL230s Gen8 ● 16 Cores ● 64Gb memory ● 500Gb local storage PBS batchsystem Server
- 13. Use As Is (Tier-1 worker node as Hypervisor): OpenNebula Conference – October 24th 2017 Madrid 13 InfiniBand Switch Ethernet Switch Eth0 (10.131.x.x)IB0 (10.132.x.x) InfiniBand network used by: ● VM network traffic (VXLAN) ● Ceph storage backend IB0 IP Gigabit Ethernet used by OpenNebula monitoring and system management. VXLANoverIPoIB Ceph storage Ceph VM1 VM2 Internet VM3 HP ProLiant SL230s Gen8 ● 16 Cores ● 64Gb memory ● 500Gb local storage Server
- 14. SSO provided by FreeIPA OpenNebula Conference – October 24th 2017 Madrid 14
- 15. Muk evolves into Grimer ● Grimer UGent HPC Public Cloud: – Running 16 hypervisors in our current testbed. – 256 CPU cores available. – 200Tb for image datastore storage (Ceph backend). – 400Tb for system datastore storage (Ceph backend). – SSO via FreeIPA. – VM network isolation with VXLAN through InfiniBand network. OpenNebula Conference – October 24th 2017 Madrid 15
- 16. PILOT USERS OpenNebula Conference – October 24th 2017 Madrid 16
- 17. Pilot Users ● Provide a virtual Docker Swarm cluster for Biomedical research – The virtual cluster is deployed automatically by the user thanks to the OneFlow service. – It provides a Docker Swarm cluster that can be resized on demand. – Also provides a Galaxy Web Frontend service to submit the jobs to Docker. ● https://www.galaxyproject.org/ – All the VMs have access to a private Ceph storage endpoint using an specific VNET and Ceph key to mount and share the data sets (~300Tb). OpenNebula Conference – October 24th 2017 Madrid 17
- 18. OpenNebula Conference – October 24th 2017 Madrid 18
- 19. Pilot Users ● Provide a Windows machine with 62Gb of memory – MaxQuant is an quantitative proteomics software for analyzing mass- spectrometric data sets. ● http://www.coxdocs.org/doku.php?id=maxquant:start – It requires a large amount of memory and it only works on Windows machines. HPC cluster, Docker or Singularity images are not an option. – The Window 10 image is now available from Grimer Cloud. – The VM provides more than 60Gb of memory (a full Grimer hypervisor) and an external 200Gb storage volume. OpenNebula Conference – October 24th 2017 Madrid 19
- 20. OpenNebula Conference – October 24th 2017 Madrid 20
- 21. LESSONS LEARNED OpenNebula Conference – October 24th 2017 Madrid 21
- 22. Some Lessons Learned – The trickiest part in our case is the network configuration due to our hardware limitations. – Fortunately the VXLAN tunnel over IPoIB works better than we have expected. Based on our Iperf benchmarks: ● VXLAN over IPoIB between 2 VMs ~6Gb/s. ● IPoIB between 2 hypervisors ~20Gb/s. – Ceph over VXLAN over IPoIB using SCSI RDMA protocol also works. – We still need to provide adhoc contextualisation scripts to mount the external Ceph storage in the VMs. – We did not use VXLAN before but it took us only 2 weeks to get the first testbed and the Galaxy use case running. OpenNebula Conference – October 24th 2017 Madrid 22
- 23. THANKS! OpenNebula Conference – October 24th 2017 Madrid 23
- 24. Álvaro Simón García HPC and Cloud systems administrator HPC UGent DICT E hpc@ugent.be www.ugent.be/hpc/en Ghent University @HPCUGent Ghent University