What we Learned About Application Resiliency When the Data Center Burned Down
0 likes181 views
The document discusses lessons learned from the destruction of OVHcloud's Strasbourg datacenter by fire, particularly focusing on application resiliency and disaster recovery strategies. It outlines the impact on services like kiwi.com and emphasizes the importance of designing systems with high availability, eliminating single points of failure, and implementing contingency plans. Key recommendations include regular testing of system resilience through 'torpedo tests' to evaluate performance under failure conditions.
![+ Architect for disaster from the start
+ Eliminate any and all single points of failure (SPOFs = bad)
+ Datacenter-aware topology & distribution; multi-datacenter replication
+ Consider sufficient excess capacity in case a whole datacenter goes down
+ Consider contingency plans for multicloud architectures or alternate standby
sites [and standby hardware]
+ Do regular chaos monkey work
Lessons Learned](https://image.slidesharecdn.com/devopsexp2021-211109183047/85/What-we-Learned-About-Application-Resiliency-When-the-Data-Center-Burned-Down-19-320.jpg)
![+ Consider running a “torpedo test” during Proof of Concepts & scalability testing
+ Take down a node, then another, then another…
+ How long does it take for a cluster to rebalance after losing a node?
+ How much of a latency/throughput hit do you take after losing a node?
+ After how many “torpedoes” does your system just “sink?” [Go non-linear]
+ How long does it take for a single node to be restored?
+ After taking n torpedoes, long does a system take to restore back to full
operations?
Consider “Torpedo Testing”](https://image.slidesharecdn.com/devopsexp2021-211109183047/85/What-we-Learned-About-Application-Resiliency-When-the-Data-Center-Burned-Down-20-320.jpg)
![A New Hope [For DevOps]](https://image.slidesharecdn.com/devopsexp2021-211109183047/85/What-we-Learned-About-Application-Resiliency-When-the-Data-Center-Burned-Down-21-320.jpg)
What we Learned About Application Resiliency When the Data Center Burned Down
- 1. What Can Be Learned About Application Resiliency When Your Datacenter Burns Down? Lessons from a real-world disaster
- 2. Peter Corless + Listen to customer stories + Write blogs & case studies + Play (and design) strategy & roleplaying games Technical Marketing Manager ScyllaDB
- 3. 10 March 2021, 0:47 AM — Disaster Strikes SBG2 datacenter of OVHcloud was entirely destroyed by fire
- 4. What would do if your datacenter was on fire?
- 5. Massive, Systemic Disaster ...as if millions of alarms triggered at once and were suddenly silenced
- 6. OVHcloud Fire OVHcloud’s Strasbourg SBG2 Datacenter engulfed in flames. (Image: SDIS du Bas Rhin)
- 7. About Kiwi.com ➔ Virtual global supercarrier ➔ Seamless travel experience ➔ Connecting “A” to “B” ➔ Virtual interlining
- 8. 8 A Database Built for High Availability Latencies briefly rise until unavailable servers are taken out of cluster 10 of 30 servers are suddenly unavailable Requests per second per server; note how some drop towards zero then blip out of existence
- 9. “This is a non-stop flight…”
- 12. OVHcloud’s Strasbourg SBG2 Datacenter the next morning. (Image via Twitter) ➔ Strasbourg datacenter impact ■ SBG2 totally consumed ■ SBG1 4 of 12 rooms gutted ■ SBG3 & SBG4 proactively taken offline ➔ Internet impact (as per Netcraft) ■ 3.6 million websites ■ 464,000 domains ■ 1 in 50 sites in all of .fr TLD Datacenter Damage Assessment
- 13. Kiwi.com’s Timeline of Fire 00:47 CET Fire breaks out in OVHcloud Strasbourg SBG2 01:12 CET Kiwi.com nodes in Strasbourg start falling off the cluster 01:15 CET All 10 Strasbourg nodes offline; traffic diverted to 2x other Kiwi.com datacenters (20 servers remaining) 02:23 CET Production operational, we manually need to tweak some services around the main database. 08:54 CET Tweaks deployed, we are fully operational
- 14. Basically...
- 15. ScyllaDB 15 + Reimagined the distributed NoSQL database + Close-to-the-hardware design, written in C++ + Open source, enterprise & DBaaS + From the creators of KVM hypervisor Winner Infoworld Technology of the Year
- 17. 17 Used across industries AdTech/MarTech Multimedia Finance/FinTech Security Ride-hailing/ Food Delivery Social Retail Travel IoT Logistics/Transportation
- 18. – Consistency Options – Eventual consistency to linearizability – Presences – 1 to 10+ datacenter replications – Volume – Multi-petabyte – Throughput – 1 billion OPS 18 – Vertical Scalability – 1 to 416 vCPUs – Horizontal Scalability – 1,000-node cluster – Availability – 1 to 10+ replicas within a datacenter – Unlimited – Cell sizes and partition width Grows with your business & your data Learn more ▸ scylladb.com
- 19. + Architect for disaster from the start + Eliminate any and all single points of failure (SPOFs = bad) + Datacenter-aware topology & distribution; multi-datacenter replication + Consider sufficient excess capacity in case a whole datacenter goes down + Consider contingency plans for multicloud architectures or alternate standby sites [and standby hardware] + Do regular chaos monkey work Lessons Learned
- 20. + Consider running a “torpedo test” during Proof of Concepts & scalability testing + Take down a node, then another, then another… + How long does it take for a cluster to rebalance after losing a node? + How much of a latency/throughput hit do you take after losing a node? + After how many “torpedoes” does your system just “sink?” [Go non-linear] + How long does it take for a single node to be restored? + After taking n torpedoes, long does a system take to restore back to full operations? Consider “Torpedo Testing”
- 21. A New Hope [For DevOps]
- 22. United States 2445 Faber St, Suite #200 Palo Alto, CA USA 94303 Israel Maskit 4 Herzliya, Israel 4673304 www.scylladb.com @scylladb Learn NoSQL for free! university.scylladb.com @petercorless