Containers on Baremetal and Preemptible VMs at CERN and SKA
1 like109 views
The document discusses the utilization of baremetal and preemptible servers for scientific projects at CERN and SKA, focusing on resource management and deployment efficiency. It highlights the advantages of using containers on baremetal for improved security and performance, as well as the challenges and lessons learned in implementing open-source tools like Magnum and OpenStack. Additionally, it proposes the development of preemptible instances to maximize resource utilization while managing project quotas in private clouds.
Containers on Baremetal and Preemptible VMs at CERN and SKA
- 1. Containers on Baremetal And Preemptible Servers At CERN and SKA
- 2. Belmiro Moreira, CERN @belmoreira John Garbutt, StackHPC @johnthetubaguy
- 6. CERN - Large Hadron Collider (LHC)
- 7. CERN: Compact Muon Solenoid (CMS)
- 8. CERN - Cloud resources status board - 11/05/2018@09:23
- 9. What is SKA? Image courtesy of CSIRO
- 12. SKA’s Science Data Processor
- 14. AlaSKA
- 15. Why Baremetal? Why Containers? ● Single security zone ● No need for virt, so target baremetal ● 30 seconds to switch ingest to Supernova, Fast radio burst, … ● Easier development and deployment cycles
- 16. Magnum with Ironic ● Magnum used extensively at CERN ● Docker Swarm and Kubernetes are supported ● Historically a separate driver for baremetal, badly maintained ● Queens moves to using Fedora Atomic for VM and baremetal
- 18. Lessons learned ● Extra network ports added after initial setup ● Updated Docker version in Fedora Atomic 27 ● Updating Atomic image with RDMA drivers was tricky ● Root disk wasn’t resized by cloud-init http://www.stackhpc.com/magnum-queens.html
- 20. Resources Utilization ● Public Clouds give the illusion of infinite capacity ○ Users pay for resources that they use ● Private Clouds ○ Resource management usually is based in project quotas ○ Prevent resources being exhausted ○ Prevent “over-committing” resources/quota ○ Manage individual projects requirements ○ Reserve resources for operations with higher priority ○ Scientific Clouds ■ Projects have different funding models ■ They expect a predefined number of resources available ■ But not always these resources are used full time
- 21. Idle Resources with quotas Project 1 Project 2 Quota utilization Quota utilization
- 22. Idle Resources with quotas
- 23. Idle Resources with quotas
- 24. Maximize Resource Utilization ● Public Clouds ○ Based on different pricing/SLA considering resource availability ○ Reserved instances vs spot-market ● Private Clouds ○ Quotas are hard limits. Leads to a reduction in resource utilization ○ Preemptible instances ■ Projects that exhausted their quota can continue to create instances ● Opportunistic workloads ● Low SLA
- 25. Preemptible Instances ● Proposal to implement Preemptible Instances into OpenStack ○ Build a prototype ○ Minimise changes required in OpenStack nova ● Starting simple ○ Use dedicated projects for Preemptible Instances ■ Avoids tagging individual instances ○ Introduce a “Reaper” service ■ Orchestrator to manage preemptible instances ● Removes preemptible instances when resources are required for non preemptible instances ● Applies a maximum TTL to preemptible instances
- 26. Workflow Instance set into PENDING state Nova notifications consumer Selects preemptible(s) instance(s) to delete Nova “Reaper” service nova-api nova-scheduler “No Valid Host” “No Valid Host” Notification 1) delete selected preemptible(s) 2) rebuild instance Reset instance ERROR state
- 27. Workflow ● The creation of a non preemptible VM fails because there aren’t available resources ● Instances that fail with “Nova Valid Host”, go to “PENDING” state instead of “ERROR” ● The Reaper service is notified and it tries to free the requested resources ○ Rebuild the instance ○ Or change instance state to “ERROR”
- 28. Current work in Preemptible Instances ● Add instance state PENDING (spec) ○ https://review.openstack.org/#/c/554212/ ● Allow rebuild instances in cell0 (spec) ○ https://review.openstack.org/#/c/554218/ ● Add scheduling notification ○ https://review.openstack.org/#/c/566470/ ● Implement instance state PENDING ○ https://review.openstack.org/#/c/566473/ ● Reaper prototype: ○ https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
- 29. Join the Scientific SIG and... Get involved! https://www.openstack.org/science/
- 30. Belmiro Moreira, CERN @belmoreira John Garbutt, StackHPC @johnthetubaguy
- 31. Join the Scientific SIG and... Get involved! https://www.openstack.org/science/