The Mushroom Cloud Effect - What happens when containers fail?
Download as PPTX, PDF0 likes177 views
The document discusses the 'mushroom cloud effect' in dynamic container environments, highlighting issues related to service dependencies, performance impacts of failing containers, and the importance of robust log management. It outlines the challenges faced by a major LATAM e-commerce company with 5,000-15,000 containers and provides insights on preventing cascading failures through automated testing and proper resource management. Recommendations include improved log management practices and preparing for peak load scenarios, such as Black Friday.
The Mushroom Cloud Effect - What happens when containers fail?
- 1. @mayralois Mushroom Cloud Effect Alois Mayr Technology Lead – Dynatrace
- 2. @mayralois Monitoring for Highly Dynamic Container Environments
- 3. @mayralois Source: http://www.schoonoart.de/ …there’s been the mushroom cloud effect oh yeah, everything screwed up
- 4. @mayralois The Mushroom Cloud Effect or What Happens When Containers Fail?
- 5. @mayralois Biggest LatAm e-commerce Company • ~ U$ 2.5 billion revenue • 4 sites: Americanas, Shoptime, Submarino, Soubarato • ~ 150 hosts across 4 regions • 5k-15k containers • 1k-3k services
- 8. @mayralois Important Aspects… • Lots of (micro-)services • Lots of communication between services • Service dependencies • Versioning and API compatibilities • Zero downtime
- 9. @mayralois Platform-related Aspects • Most often container-based • Clustered for scalability • Ephemeral containers • Resilient architecture • Cross AZ fail-overs • SDN for communication
- 10. @mayralois Deployments are no Longer Static 7:00 a.m. Low load, service running with minimum redundancy 12:00 p.m. Scaled up service during peak load with failover of problematic node 7:00 p.m. Scaled back down to lower load, move to different geolocation
- 11. @mayralois Anatomy of dynamic environments https://www.dynatrace.com/en/ruxit/
- 12. @mayralois All About (Service) Dependencies
- 13. @mayralois Failing containers… …may or may not have an (immediate) impact on service performance
- 14. @mayralois Cascading Failures Lead to a Mushroom Cloud Effect
- 15. @mayralois
- 16. @mayralois The Hungry Container Breakdown • Shared /logs partition on host • No log rotation, no archiving for app logs • No proper log management used for Docker environment • Shared /logs partition ran out of space What was the problem?
- 17. @mayralois The Hungry Container Breakdown • Container health checks failed • Orchestration killed container and rescheduled new one • Still no free space on /logs • Termination and rescheduling • /var/lib/docker ran out of space • Cluster nodes were no longer able to run any containers How the problem has evolved over time?
- 18. @mayralois The Hungry Container Breakdown • Services at the top of the graph • Increased failure rates • Lots of depending Tomcat and DB services affected How the problem affected services?
- 19. @mayralois
- 20. @mayralois The Hungry Container Breakdown Log management tools for app logs --log-driver=none|syslog Remove container --rm=true /var/lib/docker deserves its own partition How the problem could have been avoided?
- 21. @mayralois The Hungry Container Breakdown Buggy Containers May Kill Your Nodes
- 22. @mayralois Try to Break Your Clusters Early (And be Prepared for Black Friday)
- 23. @mayralois Break Your Clusters Early Massive load testing! Survive three days of pain Include everything Services, Containers, Orchestration, EC2 instances
- 24. @mayralois Testing everything 13.3k containers (+nodes) 3,451 services
- 25. @mayralois
- 26. @mayralois Automation Needed to Pinpoint the Root Cause of Cascading Failures!
- 27. @mayralois Want to learn more? Stop by our booth! G15
Editor's Notes
- #3: Who am I and what we do Dynatrace the monitoring company with a solution for highly dynamic cloud and container environments Many customers, lots of experience, production monitoring,
- #4: There are many campfire stories we could go into … but today I’m gonna talk about the Mushroom Cloud Effect Go into context here. Mushroom cloud effect we see very often in dynamic container environements with lots of microservices etc.
- #9: All the services should work as designed You need to be careful of api and service compatibilities Additional complexity through feature sets cross geo-location availability Usually designed for zero downtime Most of them apply to netflix website services
- #10: However, running such services often done with paas or paas like systems Container based One container, one service One service, multiple containers – somewhere across different hosts Containers are ephemeral, regularly killed and started Orchestration not only for scheduling but also for resilience at container level
- #12: This is an example of one of our customers – B2W the largest e-commerce provider in LatAm Left hand side: a single service in v5 provided by 5 containers – one is offline. The conainers run on separate hosts in 2 datacenters. This is the technical part Right hand side is a mess – all services communicate somehow to each other. Service dependencies and communication needs to be real-time
- #13: This is a way simpler environment that shows the flow of transactional apps.
- #16: Example of a small mushroom cloud effect Bottom: Hosts Middle part: processes / containers Top: services and applications