CloudStack upgrade best practices - Dag Sonstebo
- 2. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue A b o u t M e • Cloud Architect @ ShapeBlue • Background: • Cloud and virtualization architect with 19 years experience from the service provider, financial and manufacturing industries. • Specialize in: • Cloud infrastructure architecture and engineering. • Virtualization - VMware vSphere, Citrix XenServer, KVM. • Automation and configuration management.
- 3. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue “ShapeBlue are expert builders of public & private clouds. They are the leading global CloudStack services company.” A b o u t S h a p e B l u e
- 4. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a p e B l u e c u s t o m e r s
- 5. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a p e B l u e c u s t o m e r s
- 6. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue S h a p e B l u e c u s t o m e r s
- 7. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Upgrading CloudStack
- 8. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Upgrades - general advice • 5p’s – proper planning prevents poor performance. • Divide and conquer – don’t plan too many changes at the same time. • Do as much work as possible in advance: • Prepare all builds. • Upgrade hypervisors if required. • Ensure system VM templates are uploaded. • RTFM! • Ensure infrastructure is healthy. • Keep your users informed – and disable their access until upgraded systems healthy.
- 9. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Planning - customer impact • CloudStack GUI and API will be offline during the upgrade. • All VMs continue to run. • Virtual Routers • Will continue to function post upgrade, but can not be changed… • To upgrade these they need to be restarted • System VMs: are offline with the GUI, also need restarted.
- 10. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Lab Upgrade Testing • Production equivalent lab upgrade test – try to match: • Hardware • OS and S/W versions • Storage – at least protocol • Prepare: • VMs with production type workloads • Guest VMs with similar networking models utilised and disks attached. • Post upgrade tests: • Functional, regression and user acceptance tests. • User actions • VM, network and storage lifecycle actions • Don’t forget your integrated systems! • Rinse and repeat….
- 11. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Production DB upgrade test • Warning: carry out in a network isolated location! • Single upgraded CS server with MySQL installed. • Import production DB, then: • Cloudstack-setup-databases • Cloudstack-setup-management • Troubleshoot and fix MySQL upgrade problems, document for production run.
- 12. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Strategy – parallel builds • Standard in-place upgrade works well, but: • Doesn’t allow for technology refresh. • Rollbacks are trickier and involves snapshots, backup restores as well as loss of logs. • Overall higher risk. • Building a parallel upgrade infrastructure has some benefits: • Old CS management servers can simply be disabled. • Upgrade done on brand new VMs / servers built in advance. • Rollbacks are much simpler – switch off new servers, switch old ones back on. • The cost of this is minimal, but it does mean a few extra steps.
- 13. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Strategy – parallel builds • Management server builds • Do not seed system VM template • Do not carry out cloudstack-setup-databases • Do not run cloudstack-setup-management • MySQL • Ensure all permissions used in your existing MySQL servers is replicated on the new ones. • Create new empty databases • Configure master-slave and ensure the empty databases are replicated.
- 14. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Agent upgrades • Vmware: • Minor – but watch out for password updates. • XenServer: • Minor - watch out for changes to vhd-utils and scripts. • KVM: • Only hypervisor which requires agent update. • If large number of nodes – automate! • Test automation well in advance. • Remember to automate backup, upgrade and rollback! • Use e.g. Ansible, Puppet or Chef.
- 15. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Sacrificial infrastructure • Optional - plan and prepare sacrificial infrastructure: • For any hypervisor where agent upgrade is required , i.e. KVM. It is easier to roll back a subset of KVM hosts rather than your whole infrastructure. • Create test cases which won’t impact end users: • Accounts • VMs • Networks – with every service offering you may use. • These can be used for the initial testing before decision is made to especially do VR restarts for all end users.
- 16. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue U p g r a d e P r o c e s s u s i n g p a r a l l e l b u i l d s Step 0 RTFM! Build parallel upgrade infrastructure Upload System VM templates Step 1 Confirm system VM templates in place Step 2 Stop and disable existing CloudStack management services Step 3 Back up all your existing databases. Disable MySQL on the original DB servers Step 4 Copy and import the databases to the new MySQL master server. Step 5 Update the "host" global setting Step 6 Upgrade hypervisor s Step 7 Configure and start first managem ent server Step 8 Monitor startup Ensure all HV check in – esp. KVM! Step 9 Restart CPVM then SSVM. Restart VRs Step 10 Configure and start additional mgmt servers
- 17. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Rollback • Only roll back when all other options exhausted! • All changes since upgrade will be lost – take timing and customer access into account. Step 1 Disable new CS mgmt servers Disable new DB servers Step 2 On hypervisor infrastructure remove all VMs created since upgrade Step 3 Enable and start the original MySQL servers Enable and start the original CloudStack mgmt. servers Step 4 Restart system VMs and VRs if they don't auto start
- 18. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue Questions?
- 19. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue B y t h e w a y … . • Next CloudStack European User Group: Wednesday, April 12 @ Interoute Prague Interoute Czech s.r.o. Siemensova Street 2717/4, Prague 5, 155 00, Czech Republic https://www.eventbrite.co.uk/e/cloudstack-european-user- group-tickets-32354983627?aff=es2
- 20. C l i c k t o e d i t The Cloud Specialists ShapeBlue.com @ShapeBlue M o r e i n f o r m a t i o n • Slide deck: http://www.slideshare.net/shapeblue • Blog: http://shapeblue.com/blog http://dsonstebo.wordpress.com • Email: dag.sonstebo@shapeblue.com • Twitter: @dagsonstebo • Web: http://shapeblue.com