Leveraging chaos mesh in Astra Serverless testing
0 likes148 views
The document discusses leveraging Chaos Mesh for testing Astra Serverless, a cloud-native database offering. It outlines various types of failures, goals of chaos testing, and methods for mitigating issues, emphasizing the importance of resilience and recovery time. Additionally, it covers the setup of experiments using Chaos Mesh alongside tools like Fallout and NoSQLBench for automated testing.
Leveraging chaos mesh in Astra Serverless testing
- 1. Leveraging Chaos Mesh in Astra Serverless testing 2021-08-26 Pierre Laporte - @pingtimeout - pierre.laporte@datastax.com
- 2. Intro
- 3. Apache Cassandra™ ● Open-source (ASLv2) distributed database ● Leader-less architecture ● Linearly scalable ● Zero-downtime 3
- 4. ● Created in 2010 ● Supports and contributes to Cassandra ● Develops DSE ● Astra Streaming, powered by Pulsar ● Distributed Tests on Pulsar with Chaos Mesh & Fallout ● Astra DB, powered by Cassandra Datastax 4
- 6. ● Distributed database testing framework ● Verify data consistency ● CAP theorem ● Mostly standalone deployments Jepsen 6
- 7. ● Built on top of Jepsen ● Open-source (ASLv2) test orchestration project ● Provisions, builds, installs and configure products ● Collects test run artifacts (.log, .hdr, .csv, …) Fallout 7
- 8. ● Open-source (ASLv2) load generator ● Supports multiple protocols (CQL, Kafka, Pulsar, MongoDB, HTTP, …) ● Benchmark definition as yaml ● Lots of metrics ● Graphite/HDR historgam NoSQLBench 8
- 10. 10 Types of failures Recoverable infrastructure failures 01 Local failures (hardware induced, traffic induced, …) Disaster failures 02 Region-wide failures (storm in us-east-1, DNS in ap-northeast-2, it's always DNS, …)
- 11. Goals of chaos testing Astra Serverless ● Build confidence, trust and knowledge ● How resilient is the system? ● How big is the user impact? ● Calculate Mean Time to Recovery ● How long to fully recover from an outage? 11
- 12. Two different mitigation procedures ● Driver level - Avoid busy/dead pods ● Speculative execution policy ● Load-balancing policy ● Kubernetes level - Replace dead pods 12
- 13. Astra Serverless architecture ● Cloud-native database offering ● Architecture whitepaper 13
- 14. ● Astra DB Data Plane ● Between internal pods ● Between pods and external services Chaos Mesh usage 14
- 15. 1. Create Chaos Mesh experiments 2. Design tests in fallout with NoSQLBench 3. Automatically compute MTTR after tests All chaos tests are run weekly Automate everything 15
- 17. ● Many ideas to leverage Chaos Mesh ● Chaos tests against Cassandra 4.0 with K8ssandra ● More Chaos Mesh experiments (IOChaos, StressChaos, …) Thank you for such a useful tool ! Future work 17
- 18. We are hiring ! Pierre Laporte - @pingtimeout - pierre.laporte@datastax.com 18