Gamification of Chaos Testing
0 likes90 views
The document discusses the gamification of chaos testing and its significance in engineering, particularly within complex systems like Kubernetes. It emphasizes the importance of experimental design, the role of practice in various high-stress professions, and the necessity of pre and post-game analytics for improving performance. The author advocates for a culture where failure is normalized and viewed as a learning opportunity in the engineering process.
Gamification of Chaos Testing
- 1. Gamification of Chaos Testing Bram Vogelaar @attachmentgenie
- 2. What could this number represent ? 29.74 Seconds
- 3. That is a weird SLO??? 99.97 % Uptime
- 4. Nope, it is this guy!
- 5. Well…Actually…it is this guy
- 6. Because of this! “It takes 10K hrs to become an expert” Malcom Gladwell
- 7. What he really said! “Experts have on average spent 10K hrs learning their craft” Malcom Gladwell
- 8. Pilots and Fireman do this most of their time
- 9. They even spend a lot of time doing this
- 10. Can you honestly answer this question? Can you figure out if your platform is in an error state?
- 11. Chaos Engineering The discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production
- 13. Kubernete is so complex it grew an entire (chaos) ecosystem
- 14. Look we did some science! Published Result
- 15. How it started Experiment #1
- 16. Experimental Design Matters How representative is one gel to the next? How many replicates are significant proof? How about negative controls? Is one side of this gel representative for the other? Are the proteins separated enough? Are these really the proteins we think they are? Was this the right technique to begin with? How much sample do we have for repeats?
- 18. Game analytics help guide training
- 19. Make that sidecar work for you
- 20. What have we learned so far 1. Athletes and Musicians - Practice makes perfect - Pre and Post game analytics will point at options to adjust and win 2. Pilots/Doctors/Fireman - People respond differently to stress - Life like simulations are critical - Checklist will help you prevent making mistakes 3. Scientists - Break problems down into known tasks - Lies, Damn Lies and Statistics….and Biases - Experimental Design matters
- 21. That’s all mighty fine, but... Checklists are also incredibly boring!
- 22. So let’s translate it to IT engineering l Game Day Exercises l (Non)-Destructive Tests - Configuration Management / Infrastructure as Code l Observability - Metrics - Logs - Traces l Living in the year 3000: Breaking production on purpose
- 23. Convince Management that: Failure is normal and expected behavior
- 24. Let engineers be scientists Training == Testing == Monitoring
- 25. Get Engineers to be comfortable facing failur
- 26. When pressing $NewThing ™ into service Create “real” DoD’s, SLO’s + SLI’s and Runbooks
- 27. Large complex systems will always be in a (somewhat) degraded state