Cloud War Stories
0 likes403 views
A live TV broadcast documenting urban foxes across various cities experienced crashes on its website during peak viewership. The site was tested to handle 300 concurrent users but saw unprecedented traffic of 5000 users per second. Autoscaling was unable to provision new instances quickly enough to handle spikes lasting only 3 minutes. To prevent future crashes, the broadcasters conducted more rigorous load testing, deployed dual load balancers with health checks to redirect traffic if one failed, and moved static assets to S3 for caching.
Cloud War Stories
- 1. Cloud War Stories Crashes on Live TV
- 2. Foxes TV show Concept • A Live TV broadcast that documented the lives of urban foxes in cities around the country • The show was a nature documentary • The web site provided a second screen experience for the users allowing them to: • Ask the presenters questions • Track live foxes in real time using GPS technologies • The broadcast was happening during a public holiday so larger than average audiences where predicted • The TV audience was predicted to be ~3000 viewers of which 10% would use the second screen
- 3. Media vs the Cloud Scenario • Predicted traffic profile • 300 users a second • Load testing carried out • Passed all the tests • Announcements going live at a specific time: • Expecting short 3 min traffic spikes through out the show • 4 spikes expected during the broadcast
- 4. What testing showed us Autoscale doesn’t work in short spaces of time! Why it failed: • Autoscale reacts to cloud watch metric for load • We set a very low load level on the metric to ensure capacity was added in plenty of time • However average spin up times for EC2 instances are 3-5 mins • Our traffic spikes where predicted to last 3 mins • This meant that the autoscale would not react in time to prevent the site crashing
- 5. The Solution With public announcements you have an advantage: You know when the announcements will take place this allows you to “pre-warm” your systems. You can: • Manually scale your systems • Removing the reliance on autoscale systems • Pre-Warm your web cache • Programmatically access your site so that content is already in the caches • This stops the initial rush of users knocking your site off line
- 6. What happened on the night! • Despite all this testing the system still crashed! • During the crash we put up a holding page to explain what was happening to the users. • This also crashed!
- 7. What went wrong? • Unprecedented audience figures of 15000 viewers and 5000 hits a second at peak on the website • Last min changes to the site had invalidated caching so we had a large rush of users hitting the origin • This wasn’t our only issue however! • As more people hit the cold cache there were more calls to the ELB • This caused the ELB to autoscale • You can’t stop this happening • Each time the ELB scaled users disconnected • Causing them to hit refresh and double the problem
- 8. Lessons Learned • Despite our testing for 300 users a second we didn’t know at what point the site would crash • To address this we took the following steps: • Future tests were handled by specialist load testers • Amazon Partners • We tested to destruction • This way we could fail the site gracefully • Even though Amazon Elastic Load Balancers (ELBs) are a highly available service we took a new approach: • Dual Elastic Load Balancers Were Deployed • We used AWS Route53 to provided weighted DNS ensuring a 50/50 split of traffic • We used Route53 to provide failure detection healthchecks • As one ELB scaled the traffic went to the other and no users were disconnected • All static assets (Images, Javascript, CSS and Video) where stored in S3 away from the main platform relieving pressure in the event of an origin rush
- 10. Getting it right