Chaos Engineering Talk at DevOps Days Austin
- 1. Chaos Engineering What is it and why should I care? Matthew Brahms | Site Reliability Engineer | @matthewbrahms
- 2. For realz? Oh yeah, really! ● 2010 - Netflix created the Chaos Monkey which can wreak havoc in AWS at will deleting instances (fully customizable/controllable) -- this is OSS as of 2012 ● 2011 - Netflix creates the Simian Army--a host of chaos tools to test failure modes in your infrastructure and applications ● 2014 - the Role of Chaos Engineer is created at Netflix
- 3. What is it? Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production. - http://principlesofchaos.org/ A working definition
- 4. Alternative definition Bad things will happen to your system, no matter how well designed it is. You cannot become ignorant to it.
- 5. Do you really wanna be that person?
- 6. What you should do A fast primer on getting started with Chaos Eng (like 75 seconds short…)
- 7. It’s a discipline... - Not a process - Principled effort - Each org/user is unique in implementation and philosophy
- 8. Start with a question ● Know your distributed system well ● Whiteboard your entire system with another person ● Find domains or services where you think a failure case may exist
- 9. Be responsible... Test Case 0 Hypothesis: “Deleting/attacking our production database *may* cause a service outage?”
- 10. Good examples... “What happens if we were to unexpectedly lose a node in our cloud provider?” “What happens if CPU utilization were to be at 100% on all cores of most of our frontend servers?”
- 11. Test your ideas You need two things to proceed: 1. Means to test your ideas 2. Means to measure the outcome of your experiment
- 12. You’ll need tools... Chaos tooling is available!
- 13. Available tooling resources - Chaos toolkit (http://chaostoolkit.org/) - Netflix (https://github.com/Netflix/SimianArmy) - Gremlin (https://gremlin.com) - Chaos Engineering Hands-On Bootcamp with k8s (https://github.com/tammybutow/chaos_engineering_b ootcamp)
- 14. You gotta see it... Observability is a key tenant! Pick a tool, any tool. ...the list goes on and on.
- 15. You should see something like this... Here is a sample of what a single-core CPU attack against a Kubernetes host looks like… #somuchfun
- 16. Cool bro, now what? ● Important to codify lessons learned from failure ● If your attack fails, you still may learn something! ● Repeating this process will refine and make your attacks even better
- 17. Wanna go next level? Plan a gameday! It is a dedicated team day focused on using chaos engineering to reveal weaknesses in your system. You’ll need buy-in from all levels of your org for this. Level-up your resiliency!
- 18. Chaos Engineering vs. DevOps The Three Ways of DevOps: ✓ Systems Thinking ✓ Amplify Feedback Loops ✓ Culture of Continual Learning and Experimentation I mean seriously, should you care? Let’s check some boxes:
- 19. Where do I go from here? Good news! There is a community forming! Chaos Engineering Slack https://slofile.com/slack/chaosengineering Austin Chaos Engineering https://www.meetup.com/Austin-Chaos-En gineering-Meetup/
- 20. Thank you for your time! Any questions, feel free to reach out to me!