Rewriting DevOps - Lessons from a 15 month software rewrite
Download as PPTX, PDF2 likes536 views
The document outlines the lessons learned from a 15-month software rewrite focused on DevOps practices. It highlights the transition from traditional separate workflows of development and operations to a more integrated approach, emphasizing shared goals, common tools, and a culture of experimentation. The rewrite resulted in improved system reliability, increased deployment frequency, and better business outcomes.
Rewriting DevOps - Lessons from a 15 month software rewrite
- 1. Rewriting DevOps Lessons from a 15 month software rewrite
- 2. Matthew Boeckman Dryas.io Strategic Consulting Startup Advisor DevOps Coaching 19 years (dev)Ops Craftsy Former VP - Infrastructure @matthewboeckman
- 3. Yarn?! Founded 4/2010 200+% YoY growth through 2015
- 5. Yes, Yarn!
- 6. Yarn?! Founded 4/2010 200+% YoY growth through 2015
- 7. Before & After … in our DevOps Devs wrote code Ops wrote infrastructure Configuration Automation (puppet) Shared On-call (with escalation) Separate workstreams Dev and Ops write code and infrastructure Immutable Infrastructure Shared On-call (with less escalation) Joined workstreams 15 month agilefall (waterscrum?) rewrite
- 8. Before & After … in the app Java/Postgres Monolith Sessions Poor caching model Tightly coupled FE/BE Limited scale (micro)Services API calls Flexible caching (multi- tier) Decoupled logic Less limited scale! 15 month agilefall (waterscrum?) rewrite
- 9. Before & After … in our culture Weekly release Ceremony High risk profile Full day regression Dev and Ops work streams Functional teams CD Low risk Test automation Shared work, responsibility Objective based teams 15 month agilefall (waterscrum?) rewrite
- 10. ewww
- 12. The new stack Fastly - Content Delivery F5 & ALB - load balancing Node.js/React - FE Java/Spring - BE Packer - AMI’s Consul - service discovery Terraform - Infrastructure Postgres/RDS - database SQS/SNS/Lambda/S3 - everything else JENKINS - Orchestration & Delivery
- 13. Timeline 2010-2015 | 7/15-9/15 | 9/15 - 2/16 | 2/16 - 9/16 | 9/30/16 DevOps! Bi-Modal SRE Foundations Hugs! Phase I Phase II Phase III LAUNCH
- 14. Phase II for the Ops team
- 16. Take a chance that went badly? Try something new that exploded? Accidentally push something past the edge? Here’s an award! The enrolled sash
- 18. Phase III 2011-2015 | 7/15-9/15 | 9/15 - 2/16 | 2/16 - 9/16 | 9/30/16 Phase I Phase II Phase III LAUNCH DevOps! Bi-Modal SRE Foundations Hugs!
- 19. Silos? In my DevOps culture? Backend Ops Frontend
- 22. DevOps teams need shared perspective. Common tools
- 23. Site Reliability Engineering "Fundamentally, it's what happens when you ask a software engineer to design an operations function." Ben Treynor Sloss, founder of Google SRE
- 24. Common goals SRE is a good start
- 25. SRE Phase I (Feb-May) ● Focus on SRE stack ○ Nagios, graphite, splunk, atlassian ● Reliability metrics ○ Errors; response time ● Runbooks ● Blameless PIR everything ● Iterate
- 26. Runbooks: ● System overview ● Escalation path ● Alert descriptions ● Common failure conditions ● Known recovery procedures ● Incident history
- 27. SRE Phase II (May -> Launch -> Forever) ● Build a production environment ● Pentesting and lockdown ● Tune reliability metrics ● Load tests ● Resilience tests ● Recovery tests ● Blameless PIR everything ● Runbooks ● Iterate
- 28. Quack like a duck
- 29. Space to try DevOps teams need low-risk places to experiment and learn
- 30. Launch! ● RDS database failed in 1st hour -- zero impact ● 17 pushes in 3 days ● 50+ pushes in week 1 ● 85% automated test coverage ● Meaningful bump in all business KPI’s 60 hours of round-the-clock coverage. Zero SevOne or SevTwo incidents
- 31. Before & After … in our DevOps Devs wrote code Ops wrote infrastructure Configuration Automation (puppet) Shared On-call (with escalation) Separate workstreams Dev and Ops write code and infrastructure Immutable Infrastructure Shared On-call (with less escalation) Joined workstreams 15 month agilefall (waterscrum?) rewrite
- 32. Before & After … in the app Java/Postgres Monolith Sessions Poor caching model Tightly coupled FE/BE Limited scale (micro)Services API calls Flexible caching (multi- tier) Decoupled logic Less limited scale! 15 month agilefall (waterscrum?) rewrite
- 33. Before & After … in our culture Weekly release Ceremony High risk profile Full day regression Dev and Ops work streams Functional teams CD Low risk Test automation Shared work, responsibility Objective based teams 15 month agilefall (waterscrum?) rewrite
- 34. DevOps needs Shared culture Shared tools Shared goals and space to try