Chaos Patterns Twilio SIGNALCONF 2016
2 likes1,236 views
This document discusses the principles and benefits of chaos engineering. It describes how Netflix introduced the concept of the "Chaos Monkey" to intentionally fail components to test system resilience. The key lessons are: trust that systems can withstand failures, fixing one problem often exposes new issues, and having a culture that embraces breaking things is difficult for most organizations. It recommends starting with chaos "game days" and using testbots to validate fallback behavior and prepare for failures.
Chaos Patterns Twilio SIGNALCONF 2016
- 1. a CHAOS PATTERNS BRUCE M. WONG | @BRUCE_M_WONG LESSONS ABOUT FAILING WELL AND FAILING OFTEN
- 2. FAILURE HAPPENS BRUCE M. WONG | @BRUCE_M_WONG
- 3. “EVERYTHING FAILS ALL THE TIME” -WERNER VOGELS, CTO, AMAZON WEB SERVICES HTTP://THENEXTWEB.COM/2008/04/04/WERNER-VOGELS-EVERYTHING-FAILS-ALL-THE-TIME/ BRUCE M. WONG | @BRUCE_M_WONG
- 4. THE ORIGINAL CHAOS MONKEY CREATED BY NETFLIX CLOUD ARCHITECT, GREG ORZELL - @CHAOSSIMIA 2010 BRUCE M. WONG | @BRUCE_M_WONG HTTPS://WWW.LINKEDIN.COM/IN/GORZELL
- 5. a A STATE OF XEN AWS EC2 REBOOT, 2014 BRUCE M. WONG | @BRUCE_M_WONG
- 6. HTTP://XENBITS.XEN.ORG/XSA/ADVISORY-108.HTML HTTP://TECHBLOG.NETFLIX.COM/2014/10/A-STATE-OF-XEN-CHAOS-MONKEY-CASSANDRA.HTML HTTP://AWS.AMAZON.COM/BLOGS/AWS/EC2-MAINTENANCE-UPDATE/ 22COMPLETE NODE FAILURE 2700+ C* NODES, 218 REBOOTS 0DOWNTIME BRUCE M. WONG | @BRUCE_M_WONG
- 7. LESSON #1 : TRUST YOUR RESILIENCE BRUCE M. WONG | @BRUCE_M_WONG
- 8. SLOW IS HARD BRUCE M. WONG | @BRUCE_M_WONG
- 9. SLOW IS HARD BRUCE M. WONG | @BRUCE_M_WONG
- 10. UNBOUND QUEUES - ELASTIC ISN’T INFINITE BRUCE M. WONG | @BRUCE_M_WONG
- 11. UNBOUND QUEUES - ELASTIC ISN’T INFINITE BRUCE M. WONG | @BRUCE_M_WONG
- 12. SLOW IS HARD BRUCE M. WONG | @BRUCE_M_WONG
- 13. LATENCY MONKEY BRUCE M. WONG | @BRUCE_M_WONG
- 14. SLOW IS HARD BRUCE M. WONG | @BRUCE_M_WONG
- 15. LATENCY TESTING 2.0 - FIT HTTP://TECHBLOG.NETFLIX.COM/2014/10/FIT-FAILURE-INJECTION-TESTING.HTML BRUCE M. WONG | @BRUCE_M_WONG
- 16. SLOW IS HARD BRUCE M. WONG | @BRUCE_M_WONG
- 17. SLOW IS HARD START SLOW •ACCOUNT LEVEL •+10MS BEFORE +100MS •+1% ERRORS BEFORE +80% ERRORS DIAL IT UP •A -> D NOT * -> D BRUCE M. WONG | @BRUCE_M_WONG
- 18. LESSON # 2 : FIXING ONE FAILURE MODE EXPOSES NEW ONES BRUCE M. WONG | @BRUCE_M_WONG
- 19. WHATS SO SPECIAL ABOUT CHAOS BRUCE M. WONG | @BRUCE_M_WONG CHAOS IS A CHOICE
- 20. WHATS SO SPECIAL ABOUT CHAOS BRUCE M. WONG | @BRUCE_M_WONG OUTAGES VS CHAOS
- 21. BRUCE M. WONG | @BRUCE_M_WONG OUTAGES VS CHAOS Uncontrolled Controlled Unpredictable Scheduled Time to Detect: Minutes 0 Time to Detect Time to Resolve: ???? Time to Resolve: seconds* Analysis Time: ???? Root Cause Analysis: Intentional
- 22. MYTH OF RESILIENCE NATION’S BUSINESS, 1977 BRUCE M. WONG | @BRUCE_M_WONG
- 23. LATENCY MONKEY BRUCE M. WONG | @BRUCE_M_WONG
- 24. LESSON # 3 : THE CULTURE ASPECTS OF CHAOS ARE HARD BRUCE M. WONG | @BRUCE_M_WONG
- 25. BRUCE M. WONG | @BRUCE_M_WONG MOST ENTERPRISES HIRE PEOPLE TO FIX THINGS. NETFLIX HIRES PEOPLE TO BREAK THINGS…. …WE SHOULD EMBRACE NETFLIX'S CULTURE OF "CHAOS ENGINEERING" THROUGHOUT ORGANIZATIONS OF ALL SHAPES AND SIZES.
- 26. BRUCE M. WONG | @BRUCE_M_WONG
- 27. SEEK PROGRESS OVER PERFECTION TWILIO LEADERSHIP PRINCIPLE BRUCE M. WONG | @BRUCE_M_WONG
- 28. GAME DAYS - BENEFITS •Training New Engineers •Discover Instrumentation gaps •New Product Launches •Incident Management Practices BRUCE M. WONG | @BRUCE_M_WONG
- 29. GAME DAYS - THE SETUP •Two “on-call” teams •Separate rooms, separate slack channels •Master of Disaster •Incident Commander BRUCE M. WONG | @BRUCE_M_WONG
- 30. LEVERAGE EXISTING TESTBOTS •Functionally test fallback code •Early warning! •Existing Integrations with Telemetry, PagerDuty, Slack •Incorporate into Canary process FUTURE BRUCE M. WONG | @BRUCE_M_WONG
- 31. RECAP Lesson # 1 : Trust your resilience Lesson # 2 : Fixing one failure mode exposes new ones Lesson # 3 : The culture aspects of Chaos are HARD Get started today! Game Days are your friend - do them early and often Testbots + focus on developer productivity BRUCE M. WONG | @BRUCE_M_WONG
- 32. WHEN YOU WISH UPON A BLUE MOON BRUCE M. WONG | @BRUCE_M_WONG