[webinar] Secrets of Top-performing DevOps Teams -- at Google and Beyond
2 likes8,794 views
This document summarizes key insights from a report on DevOps performance at Google and other companies. It discusses how DevOps teams are measured based on speed and stability metrics like deployment frequency, lead time for changes, and change failure rates. It also explores how the industry has improved, with more companies achieving elite performer status, and how the use of open source tools and cloud computing correlates with better performance. Additionally, it emphasizes that perfect reliability is unrealistic and recommends using error budgets to balance speed of deployments and change against service availability targets.
[webinar] Secrets of Top-performing DevOps Teams -- at Google and Beyond
- 1. Secrets of top-performing DevOps teams at Google and beyond
- 4. Agenda ● Defining DevOps ● How to measure DevOps performance ● New insights in 2019 ● Balancing speed with stability ● How do we improve
- 5. Defining DevOps BusinessConcept Development Operations Market Agile solves this Greater agility with DevOps DevOps solves this
- 6. Measuring DevOps performance Speed Stability Deployment frequency Lead time for changes Time to restore service Change fail rate
- 7. Aspect of Software Delivery Performance* Elite High Medium Low Deployment frequency On demand (multiple deploys per day) Between once per day and once per week Between once per week and once per month Between once per month and once every six months Lead time for changes Less than one day Between one day and one week Between one week and one month Between one month and six months Time to restore service Less than one hour Less than one day* Less than one day Between one week and one month Change failure rate 0-15%b 0-15%a, b 0-15%a 46-60% Speed and stability are both possible
- 8. Adding availability to metrics for operational performance Availability is about ensuring a product or service is available and accessible Availability measures are significantly correlated with performance Availability reflects how well teams define their availability targets and learn from outages
- 9. Agenda ● Defining DevOps ● How to measure DevOps performance ● New insights in 2019 ● Balancing speed with stability ● How do we improve
- 11. Heavyweight process & control make enterprises low performers…
- 12. 20% Low performers have highest concentration of fully proprietary software Low Medium High A mix of proprietary tools, open source, and commercial off-the-shelf (COTS) software Mainly open source and COTS, heavily customized Mainly open source and COTS, with little customization Primarily COTS packaged software Primarily developed in-house and proprietary to my organization Primarily open source, heavily customized Primarily open source, with little customization 30% 34% 32% Elite 33% 17% 8% 7% 10% 14% 21% 18% 20% 8% 12% 8% 4% 6% 5% 6% 6% 7% 5% 12% 5% 12% 24% 15%
- 13. Growing role of open source in DevOps Skaffold Speed up local build, test, deploy workflow. Kubernetes For management of containerized software Istio For services management Spinnaker An open-source multi-cloud CD platform
- 14. Tekton Open-source, vendor-neutral components for running CI/CD workloads on Kubernetes to standardize implementation across the industry and increase workload portability.
- 15. “Increasingly, Tekton is becoming the universal, if you like, pipeline glue which Jenkins, Jenkins X, and possibly, hopefully one day, Spinnaker, can all use” James Strachan, CloudBees distinguished engineer Tekton Events UI Notifications Kubernetes Pipelines Tasks Jenkins X
- 16. Elite performers are 24x more likely to have met all cloud characteristics
- 17. Only 29% of respondents met all 5 characteristics of cloud computing On-demand self-service Broad network access Resource pooling Rapid elasticity Measured service
- 18. Clear wins from cloud Fast autoscaling Cost visibility Security
- 19. Full-stack serverless on Google Cloud for auto-scaling ML & AI Compute Database & Storage Data Analytics Messaging DevOps Smart assistants & chat
- 20. Cloud: Slashing cost of research at Broad Institute First GCP run Task splitting Preemptible VMs Persistent disks Algorithm tuning 0 0.25 0.5 0.75 1 Relativecost
- 21. Shopify: a story of security at scale Every day 6,000 built container images are deployed securely. Over 330,000 images stored in Container Registry
- 22. Agenda ● Defining DevOps ● How to measure DevOps performance ● New insights in 2019 ● Balancing speed with stability ● How do we improve
- 23. Ship as often as possible! Don’t break anything, ever! Why not both?
- 24. ● Humans have never created a perfectly reliable system. ● Nature has not created a perfectly reliable system. ● If systems that have evolved under competitive pressure over hundreds, or billions, of years aren’t perfectly reliable, then perhaps perfect reliability isn’t the best target? While deciding availability remember perfection is unrealistic Balancing speed with stability
- 25. Select availability level based on Balancing speed with stability Service level expected by users Does this service tie to our or customer’s revenue? Service for customers vs enterprises Paid vs free service What availability competitors provide Cost
- 26. ● An objective metric that determines how unreliable the service is allowed to be within a single quarter. ● Product management & SRE determine availability target. ● If there is error budget remaining – new releases can be pushed. Error budget helps balance availability with velocity 100 - availability target = Error budget Balancing speed with stability
- 27. Example 90% 36.5 days 9 days 3 days 95% 18.25 days 4.5 days 1.5 days 99% 3.65 days 21.6 hours 7.2 hours 99.5% 1.83 days 10.8 hours 3.6 hours 99.9% 8.76 hours 2.16 hours 43.2 minutes 99.95% 4.38 hours 1.08 hours 21.6 minutes 99.99% 52.6 minutes 12.96 minutes 4.32 minutes 99.999% 5.26 minutes 1.30 minutes 25.9 seconds Availability level Allowed unavailability window Balancing speed with stability
- 29. Agenda ● Defining DevOps ● How to measure DevOps performance ● New insights in 2019 ● Balancing speed with stability ● How do we improve
- 30. 2 research models to drive improvements Performance model Productivity model
- 31. Performance model Clear change process Heavyweight change process Code maintainability Loosely coupled architecture Monitoring Trunk-based development Deployment automation Culture of psychological safety Continuous integration Automated testing Continuous delivery Disaster recovery testing Cloud Burnout SDO Performance Software delivery performance Availability Organizational performance Industry (control) Enterprise (control) (-) (-) (-)
- 32. 3 guiding principles 1. Increase developer productivity 2. Shift left 3. Automated Compliance
- 33. Technical Practices- Demo Cloud Code GCR vulnerability scanning Binary authorization Trusted images Kubernetes Engine Cloud Build GitHub Untrusted images Audit log Local development Remote development Fixes
- 34. Driving productivity improvements Useful, easy to use tools Internal search External search Culture of psychological safety Productivity Technical debt Work recovery Burnout Code maintainability Loosely coupled architecture Monitoring Years of experience (control) (-) (-)(-)(-) (-) n.s.
- 35. Takeaway The industry continues to improve, particularly among the highest performers. DevOps matters for both commercial goals and non-commercial goals Cloud continues to be a differentiator for elite performers 100% availability is unrealistic & undesirable Error budgets can help with measuring availability
- 37. Questions?