Running Cloud Foundry for 12 months - An experience report | anynines
3 likes1,820 views
The document provides insights on running Cloud Foundry (CF), covering operational experiences, management of overcommitment settings, update processes, and hardening techniques. It emphasizes the importance of high availability services and the need for robust monitoring and failover strategies. The author shares practical tips based on their year-long experience managing a public CF environment, highlighting both its advantages and potential pitfalls.
Running Cloud Foundry for 12 months - An experience report | anynines
- 1. Running Cloud Foundry An Experience Report
- 3. • Receive an opinion about running Cloud Foundry (CF) • How to shoot your own leg with CF and overcommitment settings • How to perform CF updates • How to harden CF • Wise words about CF services
- 4. Introduction
- 6. Running a public Cloud Foundry for more than a year.
- 7. It works.
- 8. In order to run Cloud Foundry smoothly …
- 9. … refer to the package leaflet for risks and side effects and consult pivotal, cloudcredo or anynines.“
- 10. The details
- 12. Cloud Foundry OpenStack VMware Hardware
- 13. We migrated from a Rented VMware to a self-hosted OpenStack.
- 14. For more details on this: http://rh.gd/a9vmw2sos
- 16. Cloud Foundry saves investments into software development by being infrastructure agnostic.
- 17. Running Cloud Foundry. What happened.
- 18. Security Issues
- 19. • Pivotal informs partners early about issued • Usually along with fixes
- 20. OpenStack Issues
- 21. • Ext4 vs. Ext3 • DEA MTU • rsyslogd command not found
- 22. CF Gotchas
- 23. DEA evacuate & Bosh timeout race-condition
- 24. • Removing a DEA → Apps will be evacuated → DEA will be stopped • Bosh deployment will fail when evacuation takes longer than the Bosh timeout • Set your Bosh timeout accordingly!
- 26. Default overcommitment factor = 4
- 27. RAM peaks may cause random errors
- 28. • Failures during staging • Random application crashes • No meaningful log information
- 30. • Native strategy • Reduce over-commitment factor • Bosh deploy
- 32. • 8 GB VM, OC factor 4 → Announces 32 GB (V)RAM • 8 GB VM, OC factor 2 → Announces 16 GB (V)RAM • When evacuating a 32 GB (V)RAM host, another 32 GB (V)RAM host will be preferred (more free space)
- 33. Evacuation Wave
- 34. 1 GB 1 GB 1 GB 1 GB
- 35. = maximum impact on running apps!
- 36. New DEAs (OC 2) will receive apps when old DEAs (OC 4) have been stopped.
- 37. Hints
- 38. • Create 2nd resource pool for new DEAs • Deploy the 2nd resource pool before startup to stop old DEAs • (-) Needs more resources • (+) Smoother transition
- 41. • Structurally identical • Less VMs
- 42. 1. Determine new features since last release
- 43. 2. Study deployment manifest changes
- 44. 3. Apply deployment manifest changes
- 45. 4. First staging attempt
- 46. 5. Debug and Fix it!
- 47. 6. Simulate the live-upgrade
- 48. 7. Schedule maintenance on status.anynines.com
- 49. 8. Perform the upgrade and cross fingers.
- 50. CF Hardening
- 51. Accept that VMs are ephemeral
- 53. Resurrect
- 54. • Monitor VM • Re-Build VMs automatically • e.g. using Cloud Foundry Bosh • + Easy • - Takes long (minutes not seconds) • - Open Stack doesn’t release persistent disks automatically
- 55. Failover to Standby VM
- 56. Distribute CF components across availability zones
- 57. • Build disjunct networks, racks, etc. • Each disjunct zone = availability zone • Tell your IaaS about availability zones • On provision choose the AZ • Build Bosh releases accordingly
- 58. • Provide stand-by VM • Monitor VM and perform failover • IP failover using Pacemaker • + Fast failover (seconds) • - Pacemaker not easy to use (& boshify) • - Increased resource usage by stdby VM(s)
- 59. • 2 * UAA • 2 * CC • 2 * n * DEAs • 2 * Health Manager • …
- 60. UAA & CC DB = SPOF
- 61. HA Postgres
- 62. • UAA and Cloud Controller database • Single point of failure for Cloud Foundry
- 63. • Postgres not inherently clusterable > failover with standby vm • Master/slave replication • Pacemaker/corosync • IP-Failover using NIC-reattachment
- 64. That’s half way towards a PostgreSQL CF Service
- 65. • Add a V2 Service Broker • Add a provisioning logic • Provision 2-node db cluster on cf create service postgres medium-cluster
- 66. Services
- 67. “The best way to find yourself is to lose yourself in the service of others.” ― Mahatma Gandhi
- 68. Wardenized Services (community services) are cute for pet projects.
- 69. Not suitable for production.
- 70. • Implementations are outdated • One size doesn’t fit all!
- 71. No production CF without high quality services.
- 73. • Use clusterable services if possible • Implement automatic failover if not • Autoprovisioning using Bosh • Organize self-healing • (Semi-)Automatic recovery from degraded mode
- 74. Summary
- 75. • Bosh & the CF release are powerful, yet you can cut yourself. • HA Services are very necessary. • CF is ready to be used in production.
- 76. Questions?
- 77. Thank you!