Embracing Failure: Self-healing, Decentralized Resource Management for Apache CloudStack
Download as PPTX, PDF0 likes297 views
The document discusses decentralized resource management in Apache CloudStack, highlighting the importance of managing the differences between desired and actual states of resources. It emphasizes the need for consistency and reliability while orchestrating operations and includes references to various scheduling models and frameworks. Additionally, it outlines challenges related to resource failures and provides insights into self-healing and auto-scaling mechanisms.
Embracing Failure: Self-healing, Decentralized Resource Management for Apache CloudStack
- 1. Embracing Failure Self-Healing, Decentralized Resource Management for Apache CloudStack John Burwell Vice President, Software Engineering john.burwell@shapeblue.com | @john_burwell
- 2. @shapeblue #ccceu VP of Software Engineering @ ShapeBlue Member, Apache CloudStack PMC (June 2013) Ran operations and designed automated provisioning for analytic/virtualization clouds Led architectural design and server-side development of a SaaS physical security platform About Me
- 3. @shapeblue #ccceu “ShapeBlue are expert builders of public & private clouds. They are the leading global Apache CloudStack integrator & consultancy” …and we’re hiring! About ShapeBlue
- 4. @shapeblue #ccceu Bang ups and Hang Ups Can Happen to You Derive the normative operation design from failure recovery
- 5. @shapeblue #ccceu What is a Resource? Control Plane DeviceDeviceDevice (Desired State) (Actual State) Resou rce (Converges Desired with Actual State) Eventually, the desired and actual states will be co
- 6. @shapeblue #ccceu CloudStack partitions resources into zones, clusters, and pods
- 7. @shapeblue #ccceu Resource status information is stale or lost Resource definitions conflict with device state Entropy Failure Modes
- 10. @shapeblue #ccceu Orchestration operations are available and eventually consistent ... but device modifications must be consistent.
- 12. @shapeblue #ccceu Orchestration TierAP CP Automation Control Tier
- 13. @shapeblue #ccceu Desired Resource State AP CP Actual Resource State
- 14. @shapeblue #ccceu SchedulingAP CP State Convergence Resource Off Resource Sta State Transitions Hoke
- 15. @shapeblue #ccceu Simple Self-contained Locality Non-persistent Hoke Design Goals
- 16. @shapeblue #ccceu Runtime Resource View Reso urce FSM Manage ment Process Devic e Queu e State Transition 1 1 Monitor Process Reso urce Offer Reso urce Status
- 17. @shapeblue #ccceu An actor represents state and behavior Communicate by message passing — each actor has a dedicated queue or mailbox Each actor is allocated a lightweight thread — implicit lock Actor Model
- 18. @shapeblue #ccceu All resources represented in a directed, acyclic graph The root node of the graph is the region organized in the following manner: region -> zone -> pod -> cluster Each resource is a child of the partition node in which owns it Resource Graph
- 19. @shapeblue #ccceu Google’s resource scheduler Transactional shared state model enabling sophisticated, global decision making Supports both high churn and low churn workloads Multiple, pluggable schedulers working in parallel Inspiration from Omega
- 20. @shapeblue #ccceu Two level scheduler Resource Offers Pessimistic Locking Pluggable Geared towards high churn workloads Inspiration from Mesos
- 21. @shapeblue #ccceu Best Effort shared-state scheduler Multiple parallel schedulers distributed by partition Combines allocators and planners Pluggable Hybrid Scheduler
- 22. @shapeblue #ccceu Partition controllers spawn system VMs for their child partitions as need to address scheduler business and reliability guarantees Parent partition controllers monitor the health of their child partition controllers and re-spawn as necessary Auto Scaling, Self Healing
- 23. @shapeblue #ccceu Evaluate implementing the concepts in the Orleans paper to reduce the number of active actors required Determine best approach causality tracking for state transitions (e.g. version vectors) Create a library implementing these concepts to demonstrate viability and separate concerns and performance test Next Steps
- 24. @shapeblue #ccceu Gilbert, Seth & Nancy Lynch. Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. 2002. Schwarkopf, Malte; Konwinski, Andy; et. al. Omega: flexible, scalable schedulers for large compute clusters. 2013. References
- 25. @shapeblue #ccceu Hindman, Benjamin; Konwinski, Andy; et. al. Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. 2011. Bernstien, Philip; Bykov, Sergey; et. al. Orleans: Distributed Virtual Actors for Programmability and Scalability. 2014. References
Editor's Notes
- #2: Live from Alexandria, VA #cloudstackworks — thank Sebastian and the audience for their accommodation. Time crunch! Follow-up to “How to Run from a Zombie” June 2013 @ CCC Santa Clara -> processes fail slowly and quietly. Led down a rabbit hole to resource management Resource management is the core of the control backplane. It’s resilience is critical to the overall reliability of the system. Comm odity Hardware -> pulling reliability out of expensive, specialized hardware into the control plane Networks partition, disks and power supplies fail, bugs happen Derive the normative operation of the system from its failure modes Distribute the resource management function across the infrastructure to isolate failures Automatically recognize failures and recover with little to no operator intervention Churn Hoke
- #3: I have experienced “cloud” from both a developer and operations perspective.
- #4: We do engineering as well. I am presenting an important initiative for ShapeBlue to expand the workloads supported by CloudStack, and provide world-class infrastructure resilience.
- #5: Normative operation assumes it is recovering from a failure. For example, if we start the system with no data was that the result of a fresh installation or data corruption? Assuming it was data corruption means it will always work.
- #6: Highlight the transient (non-persistent) nature of a resource
- #7: These partitions become the isolation barriers for failures.
- #8: Not an exhaustive list. They are the most significant.
- #9: CloudStack must obey the rules …
- #10: Consistency is not “C” from ACID. It is atomic consistency applied to the scope of a single request/response operation sequence. “There must exist a total order on all operations such that each operation looks as if it were completed in a single instant.” Linearizable. Availability: “Every request received by the a non-failing node in the system must result in a response” Partition Tolerance: The ability of the system to operate when all messages sent between one or more nodes are lost.
- #11: Resources require operations be applied in a specific order. In order to prevent race conditions that would cause overcommitment or an invalid state, only one state transition can be applied in any given instant.
- #12: We have an apparent violation of the CAP Theorem
- #15: The flexibility of an opaque resource offer abstraction — the what not the how. We current work on the how.
- #16: I am willing sacrifice some performance for simplicity and correctness
- #17: Each resource has a finite state machine with a single queue and processing thread. We attempt to converge state. If we cannot find a convergence path, we fail fast and loudly.
- #18: No locking overhead or locks to clean up on failure
- #28: Slides will posted Slideshare shortly