Hpc to OpenStack: Our journey
5 likes702 views
The document outlines a journey of integrating HPC (High Performance Computing) into OpenStack, covering topics such as provisioning systems, configuration management, and future developments. It discusses the use of xCAT for bare metal management and the challenges faced in adopting OpenStack, while emphasizing the need for flexibility and user autonomy in HPC environments. The document concludes with future plans for leveraging OpenStack features like Heat to deploy virtual HPC environments.
Hpc to OpenStack: Our journey
- 1. HPC to OpenStack Our journey Tuesday 16th February 2016
- 2. Outline ● Key Questions ● Provisioning Systems ● Configuration Management ● Futures ● Q&A
- 3. How do we?? ● Install bare metal ● Configure networking, and base OS ● Install software ● Manage the configuration ● Make it redundant
- 4. Provisioning Systems ● Which one to choose – Foreman – Cobbler – Oscar – Rocks – xCAT – PCM – Bright – Others...
- 5. What is xCAT ● eXtreme Cluster/Cloud Administration Toolkit ● Management of clusters ● Baremetal Provisioning ● Scriptable ● Large scale management (Lightsout, remote console, distributed shell) ● Configures key services based on tables
- 6. Why we're using xCAT ● xCAT provides bare metal management and provisioning of hardware. ● Automatic Discovery ● EPL Licencing – Adaptable – No licencing cost ● Close relationship with developers ● Hardware management – Lights out management – Remote console/Serial over Lan – Remote inventory, vitals, power
- 7. Pros and Cons ● Pros – Flexible – Open-Source – HW Support – OS Support ● Cons – No GUI – No Configuration Management – Very Steep Learning Curve – Only maintained by IBM/Lenovo
- 8. How is this relevant to OpenStack ● The installation process is very similar ● The quantity of nodes are very similar ● We still need to provision nodes from bare-metal ● XCAT has an Ironic driver :)
- 9. Why NOW!!!! ● KVM performance ● SR-IOV support ● VXLAN offloading
- 10. Why OpenStack for HPC ● Users want flexibility ● Users have a choice ● Users don't depend on Administrators as much ● Not all applications are available on the system ● Sequential jobs ● Embarrassingly parallel jobs
- 11. Performance ● MVAPICH2-virt performance is approx 1-4% slower compared to native (source Mellanox booth SC15) ● 8% slower w/SR-IOV vs Native IB (source SDSC) compared to 28% 2 years ago on KVM http://mvapich.cse.ohio-state.edu/static/media/talks/slide/dk_mellanox_theatre_sc15_1.pdf
- 12. Our Dilemma ● OpenStack is so new ● Not many people know it ● Where do we start
- 13. Our Choices ● Initially (Havana - Icehouse) – CentOS – RDO – Packstack ● Middle (Icehouse – Juno) – CentOS – RDO – Crudini (openstack-config) ● Then (Juno - Liberty) – CentOS – RDO – SaltStack
- 14. Why!!! ● CentOS – Because a lot of our customers used it – Compatible with GPFS ● RDO – Natural choice after choosing CentOS – Easy way of getting all packages installed via package manager ● SaltStack – Customer introduction to the SW
- 15. How – Strike 1 ● Packstack ● Great for PoC environment ● Test any specific features ● Simple to get everything running
- 16. How – Strike 2 ● Crudini (openstack-config) ● A lot of work required to get things up and running ● Everything was scripted ● Doesn't work with multiple values for any option ● Very difficult to maintain and version control
- 17. How – Strike 3 ● SaltStack ● Created many salt formulas specific for our requirements ● Moved all our configs – OpenStack – GPFS – Icinga (Nagios) – Ganglia – 3rd -party applications – HPC specific formulas – InfiniBand
- 18. Preparing the System ● Decide the version of OpenStack ● Sync repos by using reposync from the repos – epel – RDO – CentOS ● Sample yum.conf and repo files below https://gitlab.arif-ali.co.uk/arif/openstack-lab
- 19. Preparing the xCAT ● Install xCAT RPMs ● Configure xCAT tables – nodes and their configurations – networks – switch mappings – Etc… ● Install nodes with base OS ● Use salt to configure all nodes depending on xCAT groups https://gitlab.arif-ali.co.uk/arif/openstack-lab
- 20. Future Development ● Update configs to use salt formulas ● Contribute to the salt-formula-* projects – Add any experience from field – Creating spec files for the salt-formulas ● Keystone v3 API integration – Federation support ● Use openstack-ansible ??
- 21. Finally ● Use heat to deploy virtual HPC environments ● Create private networks ● Install custom scheduler and software ● Create as many as you want https://gitlab.arif-ali.co.uk/arif/openstack-lab/tree/master/heat_templates
- 23. Q & A