Cassandra Summit 2015 - A State of Xen - Chaos Monkey & Cassandra
1 like194 views
This document discusses Netflix's experience with Cassandra and database automation during a reboot event in 2014 across all availability zones on AWS. It describes the initial stack used including Cassandra, Priam, and Atlas for managing instances. Gaps were identified in this approach and alternatives were explored, leading Netflix to decide on an asynchronous, RESTful solution. Key principles learned included using asynchronous vs synchronous communications and REST vs SSH. The future direction of this work is also briefly discussed.
Cassandra Summit 2015 - A State of Xen - Chaos Monkey & Cassandra
- 1. A State of Xen Chaos Monkey & Cassandra
- 2. Who we are Jean-Sebastien Jeannotte – JS Senior Software Engineer Platform Automation Engineering jjeannotte@netflix.com @jsjeannotte http://www.linkedin.com/in/jsjeannotte Nir Alfasi Senior Software Engineer Platform Automation Engineering alfasi@netflix.com @niralfasi http://www.linkedin.com/in/alfasin Christos Kalantzis Director of Engineering Cloud Database Engineering Cassandra MVP ckalantzis@netflix.com @chriskalan http://www.linkedin.com/in/christoskalantzis
- 8. Our stack during Re:boot 2014 C* Priam C* Priam C* Priam REST + SSH
- 9. Our stack during Re:boot 2014
- 10. Our stack during Re:boot 2014
- 11. Our stack during Re:boot 2014 C* Priam C* Priam C* Priam REST + SSH AtlasAtlasApp 1 App 2
- 12. Our stack during Re:boot 2014
- 13. Our stack during Re:boot 2014 Disappearing instance? Launch new instance All good Is the C* ring healthy? Are all instances healthy? All good Can we fix automatically? Replace bad instance All good Is there an offline maintenance? First failure? Sleep for X minutes and retry PagerDuty Is there an offline maintenance? First failure? All good Every 30 min
- 14. Our stack during Re:boot 2014 AWS Bootre: September 2014, Every AZ
- 19. New direction
- 20. New direction – What others are doing
- 21. New direction – What we decided to do
- 22. New direction – What we decided to do
- 23. New direction – What we decided to do C* Priam C* Priam C* Priam AtlasAtlasApp 1 App 2
- 24. New direction – What we learned (principles)
- 25. New direction – What we learned (principles)
- 26. New direction – What we learned (principles) Synchronous Asynchronous SSH HTTP / REST
- 27. New direction – What we learned (principles)
- 28. New direction – What we learned (principles)
- 29. What does the future look like?
- 30. What does the future look like?
- 31. What does the future look like?
- 32. Check out our https://jobs.netflix.com page for current openings
- 33. Who we are Jean-Sebastien Jeannotte – JS Senior Software Engineer Platform Automation Engineering jjeannotte@netflix.com @jsjeannotte http://www.linkedin.com/in/jsjeannotte Nir Alfasi Senior Software Engineer Platform Automation Engineering alfasi@netflix.com @niralfasi http://www.linkedin.com/in/alfasin Christos Kalantzis Director of Engineering Cloud Database Engineering Cassandra MVP ckalantzis@netflix.com @chriskalan http://www.linkedin.com/in/christoskalantzis
Editor's Notes
- #2: Building a house of cards on a solid database foundation.
- #3: Lead Cloud database Engineering for Netflix. Among other things, we offer C* as a service within Netflix. Feel free to follow me on Twitter or link up on LinkedIn.
- #4: Talk about the Simian Army - introduce simian army Netflix LOVES chaos. We love it so much that we generate it. Monkey - run in prod Kong - Exercice We run it on most of Netflix services, and even on C*
- #5: Talk about the Simian Army - introduce simian army Netflix LOVES chaos. We love it so much that we generate it. Monkey - run in prod Kong - Exercice We run it on most of Netflix services, and even on C*
- #6: Talk about the Simian Army - introduce simian army Netflix LOVES chaos. We love it so much that we generate it. Monkey - run in prod Kong - Exercice We run it on most of Netflix services, and even on C*
- #7: CDE has Chaos Monkey enabled on our C* clusters Maximum 1 node per day, during business hours Our Healthcheck dectects the missing instance and replaces it
- #8: 218 C* nodes rebooted 22 nodes didn’t start and were automatically terminated by AWS internal healthcheck Our heathcheck identified the missing nodes and automatically remediated the issue 0 downtime
- #9: - Bunch of Python/Shell scripts - Jenkins as job scheduler (HC, node-replacements, repairs, upgrades and etc) - On C* nodes: C* + Priam - Is something missing? Monitoring? OpsCenter?
- #10: - Why not OpsCenter? - Didn’t exist when Netflix started using C* - Redundant in our stack
- #11: ( continuation on why not OpsCenter) - change slide according to christos's feedback - Atlas is already a very powerful metrics and alerting tool, and our metric systems add non-C* related metrics (App metrics for example) that help in correlation. Alerts can be a combination of C* and App metrics. - How it behaved during the Re:boot - How did the healthcheck behave, how does it work and react to Chaos Monkey
- #12: ( continuation on why not OpsCenter) Atlas is already a very powerful metrics and alerting tool, and our metric systems add non-C* related metrics (App metrics for example) that help in correlation.
- #13: ( continuation on why not OpsCenter) Alerts can be a combination of C* and App metrics.
- #14: Healthcheck flow 2 scenarios are automatically remediated
- #15: How did the healthcheck behave during Re:boot
- #16: HC - Big monolith About 100k lines of Python/Bash scripts Hard to maintain
- #17: Lack of chaining (statefulness: if this job failed run that, else…) Stateless Lack of native support for TRIGGERING jobs based on events, like listening to SQS queues
- #18: High Availability: The Jenkins master node is a Single Point of Failure Long running processes may crash due to a transient connection issue between the slave & the master
- #19: High Availability: The Jenkins master node is a Single Point of Failure Long running processes may crash due to a transient connection issue between the slave & the master
- #20: What we learned, and what we decided to focus on (Principles)
- #21: What others are doing: Facebook (FBAR) / LinkedIn (Nurse) / DropBox (Naoru)
- #22: Do our own or adopt existing solution? We started with our own POC, then we decided to go with Stackstorm- event-driven automation platform Facilitated Troubleshooting/Event handling Automated remediation (Discovery example)
- #23: Do our own or adopt existing solution? We started with our own POC, then we decided to go with Stackstorm- event-driven automation platform Facilitated Troubleshooting/Event handling Automated remediation (Discovery example)
- #24: What we decided to do: new env SackStorm-desc (rules/actions…) Example of the Disk Space Alert gap recap
- #25: Idempotence (make a stateless system feels like a stateful system) Automation tools need to assure that you reach a certain state Example: Downloading the C* tarball: First, check the nodetool version
- #26: K.I.S.S. - “Simplicity is the ultimate sophistication” (Example: Resumable repairs - make more concise)
- #27: Prefer HTTP over SSH and Async over Sync
- #28: Retries with Timeouts and exponential back-off
- #29: Serving-fallbacks Example: Dynamic property service with hard-coded defaults Netflix personalized recommendations falling back to default recommendations
- #31: Audit trail: use logstash to index data into Elasticsearch for Trend Analysis - Talk about the fact that we already use LogStash @ Netflix, but we want to plug it into our automated remediation system
- #32: Metadata / Statistics / Long term metrics Use Trend Analysis to be proactive instead of reactive: Disk usage to predict when we need to increase the cluster size with automated resizing
- #34: Lead Cloud database Engineering for Netflix. Among other things, we offer C* as a service within Netflix. Feel free to follow me on Twitter or link up on LinkedIn.