Paging, Alerting, Chaos Eng Overview
- 2. What qualifies as an incident?
- 3. What should happen in an incident?
- 5. What tools would we use to gather data?
- 6. How reliable/accurate would this information be?
- 7. What’s the acceptable time to be alerted/respond?
- 8. An Initial Framework for Discussing SEVs
- 9. What is a SEV? SEV is a term used to refer to an incident, it is derived from the word severity.
- 10. Common types of SEV - Availability Drop - Product Issue / Feature Broken - Data Loss - Security Risk - etc.
- 11. SEV Levels
- 12. Any SEV which involves a loss of customer data should be classified as SEV0.
- 13. Calculate Critical Uptime in 9’s https://uptime.is/
- 14. What uptime do you think we could safely publish? Would we be okay telling our customers/vendors that number?
- 15. SEV Terminology
- 16. Lifecycle of a SEV event
- 17. How to measure a SEV? % loss * outage duration
- 19. How do we maintain combat effectiveness during a SEV?
- 20. Incident Manager On-Call (IMOC) - Should be a small rotation of Engineering Leaders - Only one person is on-call in this role at any point in time - These people should possess a wide knowledge of services and engineering teams - Will be our version of Air-Traffic-Control for the SEV, ensuring different people working on the SEV are organized and working coherently as a unit!
- 21. Tech Lead On-Call (TLOC) - This would be the engineer driving resolution of the SEV - Should have deep knowledge of specific domain of knowledge; be a SME (Subject Matter Expert) - Should have a deep knowledge of upstream and downstream dependencies
- 22. What we need to define to have these roles: - IMOC runbook/guide - Designate a Primary and Secondary IMOC at all times - Escalation should be automatic - Monthly sync for all IMOC and TLOC - Way to quickly triage what systems are effected/find root cause - How would we do this? - How do we record / document SEV’s? - Google Form? Git repo? Suggestions?? - SEV naming convention
- 23. What happens when we don’t meet our uptime requirements?!?
- 25. Technical Issues ● Dependency Failure ● Cloud Provider Region/Zone Failure ● Provider Failure ● Connectivity Issues ● Power issues (our local office power affects AWS RDS!) ● DNS outage/latency ● Misconfiguration of machines/docker images ● Software Bugs ● Corrupt/unavailable backups
- 26. Cultural Issues ● Lack of knowledge sharing ● Lack of knowledge handover ● Lack of on-call training ● Lack of chaos engineering ● Lack of a high severity incident management program ● Lack of documentation and playbooks ● Lack of alerts and pages ● Lack of effective alerting thresholds ● Lack of backup strategy
- 27. How do we prevent SEVs from repeating? ● Combination of: ○ Record outages ○ Correlate failures ○ Track SEVs
- 29. What if we could break things safely!? What lessons/data could we gather?
- 30. Chaos Engineering...yes, it is a real thing! ● 2010 - Netflix created the Chaos Monkey which can wreak havoc in AWS at will deleting instances (but fully customizable/controllable) -- this is OSS as of 2012 ● 2011 - Netflix creates the Simian Army--a host of chaos tools to test failure modes in your infrastructure and applications ● 2014 - the Role of Chaos Engineer is created at Netflix
- 31. Principles of Chaos Engineering
- 32. Can we do this?
- 34. And we all get to “share the pain” with our new tool PagerDuty...
- 35. Credits - https://www.gremlin.com/community/tutorials/how-to-establish-a-high-severity-inciden t-management-program/ - https://www.gremlin.com/community/tutorials/chaos-engineering-the-history-principles -and-practice/ - https://www.gremlin.com/the-discipline-of-chaos-engineering/ - https://github.com/tammybutow/chaos_engineering_bootcamp - https://www.usenix.org/conference/srecon17americas/program/presentation/andrus They did a cool workshop about Chaos Engineering with hands-on labs at SREcon this year. If you like this notion of chaos, more of us should go next year! #notjustforSRE