Introduction to Chaos Engineering with Microsoft Azure
1 like371 views
The document introduces chaos engineering as a proactive approach to identify potential weaknesses in complex systems, particularly within Azure environments. It outlines the expenses related to downtime and offers a methodology for conducting chaos experiments, including the necessity of automation for continuous testing. Additionally, it highlights several experiments conducted with Azure Kubernetes Service to validate hypotheses related to pod recovery, autoscaling, latency, and storage constraints.
Introduction to Chaos Engineering with Microsoft Azure
- 1. Introduction to Chaos Engineering with Microsoft Azure Ana Margarita Medina Gremlin, Sr. Chaos Engineer ana@gremlin.com @ana_m_medina
- 2. Our businesses, health, and safety rely on applications and systems that will fail 2 @ana_m_medina
- 4. Failures are inherent to complex systems and will cause downtime unless tested for. @ana_m_medina
- 5. Application Code Dependencies Configuration Infrastructure People and Processes Traditional testing only covers a small portion of the software stack Traditional Testing is not enough anymore.
- 6. Cost = R + E + C + ( B + A ) During the Outage R = Revenue Lost E = Employee Productivity After the Outage C = Customer Chargebacks (SLA Breaches) Unquantifiable B = Brand Defamation A = Employee Attrition The average company loses $300,000/hour of downtime Measuring the Cost of Downtime @ana_m_medina
- 8. 8 Gremlin, Sr. Chaos Engineer Developer since 2007 SRE/Chaos Engineer since 2016 Ana Margarita Medina @ana_m_medina @ana_m_medina
- 10. Chaos Engineering is thoughtful, planned experiments designed to reveal weakness in our systems. @ana_m_medina
- 12. How to do Chaos Engineering? 1. Observe Your System 2. Baseline your metrics (set SLOs/SLIs per service) 3. Form a Hypothesis with Abort Conditions 4. Define Blast Radius 5. Run Experiment 6. Analyze Results 7. Expand Scope and Re-Test 8. Share Results @ana_m_medina
- 14. 14 We want to, but...
- 15. Excuse: We don’t have time.
- 16. Excuse: We don’t need to break things. They break on their own!
- 17. Excuse: We don’t know how to get started.
- 19. We test proactively, instead of waiting for an outage. @ana_m_medina
- 20. How do we do Chaos Engineering when working on Azure? @ana_m_medina
- 22. Experiment 1: Pod Restart/Replication Hypothesis If a Pod shuts down, K8s automatically restarts the Pod or deploys a replica. Procedure Use a shutdown Gremlin to stop the pod. Observation Pod enters shutdown state, and Kubernetes immediately restarts it. The application returns to a healthy state. Conclusion K8s can successfully recover from a Pod failure without significant interruption to users.
- 23. Experiment 2: Validating Autoscaling Hypothesis Using excessive CPU triggers the Cluster Autoscaler to provision a new node. Procedure Use a CPU Gremlin to consume 50% of CPU on all nodes for 2 minutes. Observation The Cluster Autoscaler provisions a new node after one minute. Meanwhile, new Pods appear in the Pending state. Conclusion Our autoscaling policy added a node and avoided evicting Pods due to resource exhaustion.
- 24. Experiment 3: Latency and Networking Hypothesis Increasing the latency of a downstream Pod will cause poor performance upstream. Procedure Use a latency Gremlin to add 300 ms of latency to the product catalog Pod, then use ApacheBench to time requests. Observation Requests are notably delayed. Conclusion We need to optimize network throughput or develop a load balancing strategy.
- 25. Experiment 4: Storage Constraints Hypothesis Pods with defined storage limits do not exceed those limits. Procedure Use a disk Gremlin to consume 90% of storage on a redis Pod. Observation Kubernetes identifies disk pressure and starts evicting Pods. Conclusion We need to increase our storage limits, review disk pressure thresholds, or set resource limits/quotas.
- 31. Latency - 1000ms app:mhc-front 120 seconds X What would you case to HALT this experiment? Demo Environment
- 32. Latency - 1000ms app:mhc-front 120 seconds X ● User is unable to use site ● Data Loss Demo Environment
- 35. Why Automate? - Prevent Regression - Prevent Human Error - Reduce time spent manually running tests - Reduce Toil
- 36. Chaos Engineering Automation - Status Checks - Scheduling - API Calls - SDK Implementations - CI/CD
- 44. observe
- 45. If needed, manual step available
- 46. What does Chaos Engineering Maturity look like? - The end goal is automated Chaos Engineering Experiments ran across all environments for all critical services continuously.
- 47. About Gremlin The leading enterprise Chaos Engineering platform Founded in 2015 by Chaos Engineering pioneers from Amazon and Netflix. Industry Leaders Enterprise-grade security, scale, control, and support. Built for Enterprise Guided expertise throughout the customer journey. Trusted Partner
- 48. Mission Build a more reliable internet.
- 49. Trusted By Teams Worldwide
- 50. Intuitive interface and well-documented API Simple Safely halt and roll back any experiment Safe SOC II Certified RBAC, MFA, SSO Secure Cloud Native Guided Breadth Why Gremlin Runs in all cloud environments - AWS, Azure, GCP Recommended attacks and scenarios Supports Linux, Windows, Kubernetes, and more
- 51. Failure happens. Run Chaos Engineering experiments on everything. @ana_m_medina
- 52. Failures are inherent to complex systems and will cause downtime unless tested for. @ana_m_medina
- 53. Interested in hands-on learning with Azure? Chaos Engineering on Azure March 31, 2021 10AM PST • 1PM EST gremlin.com/bootcamps Interactive Learning @datadoghq | @gremlininc
- 54. Join the Chaos Engineering Community: gremlin.com/slack @ana_m_medina
- 57. 61 Ana Margarita Medina Sr. Chaos Engineer, Gremlin ana@gremlin.com @ana_m_medina Thank You! Try Gremlin Free: go.gremlin.com/ana Free Stickers: gremlin.com/talk/chaos-azure