Amazon cloud failure
0 likes598 views
An incorrect traffic shift onto the lower capacity EBS network in one availability zone caused many nodes to become isolated and lose data replication. When the traffic shift was rolled back, the nodes began an intensive search for available storage space to re-mirror data, exhausting the cluster's free capacity. This led to a re-mirroring storm that left many volumes stuck. The overloaded EBS cluster could no longer service API requests from the control plane, causing thread starvation that disrupted other availability zones and regions. The root causes were nodes not backing off replication requests aggressively enough when failing to find space, and a race condition bug exacerbating node failures during large numbers of replication requests.
Amazon cloud failure
- 1. Amazon AWS DisruptionSuhas A. Kelkar, Director, Incubator TeamMay 2nd, 2011
- 2. 5/6/20112What Happened…Easter Weekend AWS Disruption…Affected region
- 3. Amazon BasicsImage Credit: DongWoo Lee from blog.edog.net
- 5. Geographically five regions: US East (N Virginia), US West (N California), EU (Ireland), APAC (Singapore) and APAC (Tokyo)
- 6. AWS EC2 SLA 99.95% availability for each region
- 7. Contains one or more Availability Zones
- 9. Distinct locations engineered to be insulated from failures in other availability zones in the same region
- 10. EBS (Elastic Block Storage)
- 11. Create storage volumes 1GB to 1TB
- 12. One can create a file system on top of EBS volumes
- 13. EBS volumes are kept in a Availability Zone and can be attached to instances also in that same zone
- 14. Automatically replicated within same Availability Zone
- 15. Control plane services coordinate user requests and propagate them to EBS clustersEBS ArchitectureRegions1..5Control Plane ServicesAvailability Zone 1..nEBS Cluster 1EBS Cluster nSecondary Replication Low Bandwidth NetworkPrimary High Bandwidth Network....Node 1Node 2Node n....5/6/20115
- 16. 5/6/20116Amazon Cloud Disruption Post Mortem The Trigger
- 17. Incorrect traffic shift onto the lower capacity EBS network
- 18. Many nodes in the affected AZ got completely isolated and lost connection to their replicas
- 20. After rolling back incorrect traffic shift, the previously isolated nodes now began searching the EBS cluster for available server space so they could re-mirror data
- 21. Free capacity of cluster was soon exhausted leaving many nodes stuck in a loop searching for free space
- 22. This led to a re-mirroring storm where a large number of volumes were effectively “stuck” while the nodes searched on and on
- 23. Why did this affect other AZs
- 24. The EBS cluster became unable to service “create volume” API requests
- 25. This caused thread starvation in EBS control plane affecting service to other AZs5/6/20117Amazon Cloud Disruption Post Mortem (contd.) The 2 major factors at the root of this problem
- 26. Nodes failing to find new nodes did not back off aggressively enough
- 27. Race condition in the code (bug) that caused nodes to fail incorrectly when they were closing a large number of replication requests.Failure walk-throughControl Plane ServicesThreadsAvailability Zone 11. Traffic usually through primary high bandwidth networkEBS Cluster 12. Traffic incorrectly shifted (manual error) to low band width secondary networkSecondary Replication Low Bandwidth Network3. This caused congestion in secondary networkPrimary High Bandwidth Network4. Nodes assumed replica destination has failed5. Mistake quickly realized and Traffic shift rolled backNode 1Node 1Node 2Node nNode 26. Re-mirroring storm (due to previous node isolation)....7. Free space runs out , nodes get stuck in a loop, volumes get stuckNode n8. API requests from control plane get stuck holding up threads in the control planeServices to other AZs affected due to thread starvation in the control plane. Control Plane essentially experienced a distributed DoS attack!5/6/20118
- 28. 5/6/20119Lessons learnt Propagation of the Problem
- 29. Other AWS Services such as the Relational Database Service (RDS) and EC2 instances rely on the EBS Control Plane for their data volume needs.
- 30. RDS in particular uses multiple EBS volumes simultaneously.
- 31. Hence this issue that started with one AZ in one region quickly affected other regions
- 32. Lessons Learnt
- 33. Applications should not rely on a single Region or AZs
- 34. Appropriate Policies and Change Management Process (requiring manual approvals for risky changes) needs to be implemented
- 35. Automation instead of manual changes would have helped prevent errors
- 36. Modify the re-mirroring search algorithm to back off more aggressively in case of a large scale interruption
- 37. Making highly-reliable multi-AZ deployments easy to design and operate so that the customer adoption is accelerated
- 38. Finally, having a fail over hybridorheterogeneous environment would have helped.