Stress Test & Chaos Engineering
1 like386 views
The document discusses stress testing and chaos engineering techniques to test the reliability, availability, and scalability of systems and services. It recommends starting with functional requirements and productivity, then stress testing services with increasing loads to identify breaking points and understand metrics. Fallback mechanisms need continuous testing to avoid outages. Chaos engineering techniques intentionally introduce failures to test infrastructure resilience to different failures like instance, availability zone, or region outages. Stress testing and chaos engineering are important to ensure systems can withstand failures and growing usage.
Stress Test & Chaos Engineering
- 1. Stress Test & Chaos Engineering Diego Pacheco
- 3. Every engineer + manager think about ❏ FRPs(Functional Requirements) ~ Features ❏ Time / Productivity ❏ Business Logic that works ❏ But is it all...?
- 4. Scalability + Availability + Reliability ❏ As the business grows would the code continue working? ❏ Would the user experience be the same(getting slow)? ❏ Would be good for some users(p50) but few users really might have a bad experience (p99.9 & p99.99). ❏ Does the user trust the system? Lack of think About this 3 disciplines could really destroy Your brand really fast.
- 6. 8 Fallacies and Actions...
- 9. The Rise and Fall of fallbacks ❏ Hystrix ❏ Spring Cloud -> Resilience4J ❏ Fallback Issues: ❏ Hard to Tests ❏ Fallbacks fail ❏ Lack of continuous testing ❏ Fallbacks can make outage even worst ❏ Amazon Philosophy -> focus in code more resilient.
- 10. Erlang | Akka | Amazon Philosophy
- 12. How to do Proper Stress / Load Testing? ❏ Have Plan ❏ What Service to Test? Why? ❏ Select Endpoints to test (don't test them all) ❏ Have Expectations in sense of Latency | Requests to Handle ❏ Know where your service break. Figure it out why. ❏ Test using batteries: 1,5,10,50,100,1k,2k,5k,10k,50k,100k,1M,100M... ❏ You must have observability. Dedicated Env is a must as well. ❏ Understand your metrics(which ones per service) ❏ Automate Stress Tests in your build pineline ❏ Have platform: It could be a jenkins job + scripts.
- 13. Stress / Load Testing with Gatling https://gist.github.com/diegopacheco/faf7ceb2496e4ebdaded
- 14. Stress / Load Testing with Gatling
- 15. Stress / Load Testing with Gatling
- 16. Stress / Load Testing with Gatling docker run diegopacheco/time-microservice
- 17. Stress / Load Testing with Gatling ./gradlew gatlingRun-com.github.diegopacheco.gatling.microservices.st.StressTest -DGATLING_URL="http://172.17.0.2:8080"
- 18. Stress / Load Testing with Gatling
- 19. Stress / Load Testing with Gatling https://gatling.io/docs/current/cheat-sheet/
- 20. Chaos
- 21. Chaos
- 22. Chaos ❏ Test your Infrastructure ❏ All ASG in place? ❏ Does the failover to other: Instance, AZ, Region works? ❏ Test your clusters: ❏ SQL | NoSQL | NewSQL ❏ Test your microservices downstream dependencies ❏ Timeouts ❏ Retries | Exponential backoff + Jitter ❏ Chaos Inside a Box ❏ DISK, CPU, Memory, Metadata...
- 23. Chaos
- 24. Chaos
- 25. Chaos
- 26. Chaos
- 27. Chaos
- 28. Chaos
- 32. Chaos
- 33. Exercises Constraints 1. Stress Tests need to be written in Scala 2. You need to use Gatling 1. Using previous exercises or time-timecroservice image make the application run in kubernetes. 2. Create a stress test with Gatling 3. Create chaos with kubernetes killing PODS and make sure app still works and gatling tests don't fail.
- 34. Stress Test & Chaos Engineering Diego Pacheco