The MTTR Chronicles: Evolution of SRE Self Service Operations Platform
- 1. The MTTR Chronicles Evolution of SRE Self Service Operations Platform SRE Director: Jason Wik Senior SRE: Jayan Kuttagupthan SRE: Shubham Patil SRECon Asia 14 June 2019
- 2. Agenda 2 SRE @ VMC on AWS Square One βeta Connecting the Dots Virtual War Room Around the Corner Takeaways
- 3. 3 SRE @ VMC on AWS
- 5. 5 SRE is focused on ensuring services are homogeneous as they scale Complexity and variability increase risk to the SLAs SaaS vs Individual Multi Environment Infrastructure at Scale Storage SaaS Scales Horizontally Compute Networking
- 6. 6 SaaS Service continues to scale horizontally with the number of customers Each customer environment scales vertically SaaS vs Individual Multi Environment Infrastructure at Scale Each customer environmentis provisioned as a homogenous environment. Multi VPN DX CPU Bound Storage BoundHCX DraaSAWS DX HCX HorizonsMulti-Az Scaled up Mgt. VMsCustomer Environment Each environment quickly deviates as utilization, features, network access, and resource bounds vary significantly.
- 7. 7 MTTR directly correlates to meeting your SLAs and SLOs Information and automation is Key The right information at the specific time The MTTR Challenge
- 8. 8 An Extensible, Dynamic, and Collaborative platform to reduce MTTR and improve operational efficiency for unique and constantly changing environments North Star “ ”
- 9. 9 Square One
- 10. 10 Information silos High Context Switch Cross team Collaboration Time consuming Postmortem & RCA Longer MTTR == More Impact to the Customer High Operational Toil Cumbersome impact assessment Past learnings not used effectively Problems Effects Problems at Hand
- 12. 12 Challenges What data should be displayed? How to leverage this for users beyond SRE? What are the information sources? How should the data be displayed?
- 13. 13 βeta: Services under a roof How to correlate data & make it work @ scale?
- 14. 14 Connecting the Dots: Single Pane of Glass
- 15. 15 Cause, Symptom & Action at a single place Display appropriate data Intuitive UI : Realtime & Responsive Integration with BI* & ticketing service Increase in user adoption. Easier onboarding Easier correlation & minimal context switch Data Driven Decisions & Trend Analysis ~20% improvement in MTTR (on initial release !!) *BI = Business Intelligence Changes Results Connecting the Dots ... How to collaborate better?
- 17. 17 Central Place for Incident Remediation More focus on development One integrated platform Effective Collaboration with Service Owners Results Agility, Consistency and Control @ SCALE
- 18. 18 Avoid information Silos Do not reinvent tools Display what’s necessary Automate ! Integrate ! Collaborate ! Learn from past incidents Cap operational work More Code, Less Toil ! "Build the platform for your Organization!" Takeaways to Empower your SRE