A One-Stop Solution for Puppet and OpenStack
1 like8,257 views
CERN uses Puppet and OpenStack to manage its large HPC infrastructure consisting of 270 hypervisors and 2900 VMs. It aims to expand this cloud infrastructure to 15,000 hypervisors and 150,000-200,000 VMs by 2015, potentially across multiple sites. Puppet modules define configurations for common use cases like OpenStack hypervisors or Hadoop nodes. Foreman is used for lifecycle management and as an external node classifier to tell Puppet masters a node's configuration. Secrets are secured using techniques like Hiera encryption with GPG.
A One-Stop Solution for Puppet and OpenStack
- 1. A one stop solution for Puppet and Openstack Daniel Lobato Garcia daniel.lobato.garcia@cern,ch @eLobatoss
- 3. What is CERN Between Geneva and the Jura mountains, straddling the Swiss- French border Mission: learn what is the universe made of and how does it work? 3
- 4. Fundamental questions in physics Why do particles have mass? What is 96% of the universe made of? Why isn’t there anti-matter in the universe? What was the state of matter after the Big Bang? 4
- 9. Current status • 270 Openstack hypervisors • 2900 virtual machines • 300 users • 14 Puppet masters • 6 Foreman backend nodes • Some production services migrating to our cloud – early birds 9
- 10. Goals • Ramp up to 15K hypervisors – 150-200K vms in 2015 • Multi-site (Hungary) 10
- 11. 8/12/2013 Document reference 11
- 12. 8/12/2013 Document reference 12
- 13. Why? • Unnecessary homebrew stack of tools • Shift to cloud standards with minimal customizations • High turnover – can’t teach new tools 13
- 14. Why? • Symbiotic relationship with the community 14
- 15. Openstack? • Modular IaaS free open source project • APIs ~compatible with those of Amazon 15
- 16. Openstack Nova (compute) Cloud fabric controller 16
- 17. Openstack Keystone (Identity) RBAC Integrated with LDAP Multiple auth* methods 17
- 18. Openstack Glance (Images) Discovery, registration, delivery of images 18
- 20. Modules • Puppet definitions for every use case you can imagine. • Dynamic environments • Hadoop node • Openstack hypervisor • … you name it 20
- 21. 21 Workflow..?
- 22. Modules and Git • Manifests and hieradata are version controlled 22
- 25. 25 Git workflow
- 26. 26 Git workflow Jens ‘Puppetfiles’ Separate repositories Makes environments and creates them on the masters
- 27. Foreman • Lifecycle management tool for VMs and physical servers • External Node Classifier – tells the puppet master what a node should look like 27
- 28. 28
- 29. 29
- 30. Power operations & Foreman 8/12/2013 Document reference 30 Foreman Proxy Physical boxIPMI Physical box IPMI Physical box IPMI VMVMVM Openstack Nova API
- 31. Openstack VM creation 8/12/2013 Document reference 31
- 32. Openstack VM creation 8/12/2013 Document reference 32
- 33. Openstack VM creation 8/12/2013 Document reference 33
- 34. Scalability experiences • Split up services • Puppet – critical vs non critical 34 12 backend nodes Batch 4 backend nodes Interactive
- 35. Scalability experiences • Foreman – split into different services 35 ENC Reports processing UI/API Load balancer 9443 – UI/API 9444 – Reports 9445 – ENC …
- 36. Scalability experiences • Autoscale via alarms (Heat) • Define situations (i.e: load threshold..) • Spin up VMs as needed 36
- 39. 39
- 41. Secrets provisioning (naïve) • Use case: provision a db password 41
- 42. Secrets provisioning (hiera-gpg) • Use case: provision a db password 42
- 43. Secrets provisioning (hack) • Use case: provision a db password 43
- 44. Secrets provisioning •Masters need not read secrets 44