Why we don’t use the Term DevOps: the Journey to a Product Mindset - DevOpsCon Berlin 2019
3 likes1,448 views
The document discusses Zalando's perspective on the term 'DevOps' and its evolution, emphasizing that it reflects a culture of collaboration between development and operations teams rather than a specific role or title. It highlights the importance of empowered, cross-functional teams in product development and the need for transparency and effective communication to address common challenges. Additionally, it outlines various practices, tools, and metrics that support efficient product delivery and developer satisfaction within the organization.
![59
STACKSET CRD
kind: StackSet
...
spec:
ingress:
hosts: ["foo.example.org"]
backendPort: 8080
stackLifecycle:
scaledownTTLSeconds: 1800
limit: 5
stackTemplate:
spec:
podTemplate:
...
github.com/zalando-incubator/stackset-controller](https://image.slidesharecdn.com/2019-06-12whywedontusethetermdevops-devopsconberlin-190613151013/85/Why-we-don-t-use-the-Term-DevOps-the-Journey-to-a-Product-Mindset-DevOpsCon-Berlin-2019-59-320.jpg)
Why we don’t use the Term DevOps: the Journey to a Product Mindset - DevOpsCon Berlin 2019
- 1. Why we don't use the term DevOps 2019-06-12 HENNING JACOBS @try_except_ MOSTAFA NAGEEB @MostafaNageeb
- 2. 2 ZALANDO AT A GLANCE ~ 5.4billion EUR revenue 2018 > 250 million visits per month > 15.000 employees in Europe > 79% of visits via mobile devices > 26 million active customers > 300.000 product choices ~ 2.000 brands 17 countries
- 3. 3 THE BIRTH OF DEVOPS - 2009 “10+ Deploys per Day: Dev and Ops Cooperation at Flickr.” DevOps Days Belgium https://devops.com/the-origins-of-devops-whats-in-a-name/
- 7. 7 THE PHOENIX PROJECT - 2013 "The Three Ways" 1. Systems Thinking 2. Amplify Feedback Loops 3. Culture of Continual Experimentation And Learning
- 10. 10 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 12. 12
- 15. 15 DEV + OPS ⇒ DEVOPS Devs "DevOps" Team Ops
- 16. 16 DEV + OPS ⇒ ENGINEERING Devs "DevOps" Team Ops Engineering Team
- 17. 17 PRODUCT DEVELOPMENT TEAMS "Great products emerge from empowered, fully cross-functional teams." - John Cutler, 2019 https://amplitude.com/blog/on-being-product-led
- 18. 18 DEVOPS IS It is a culture, in which people work together to improve the product delivery cycle.
- 19. 19 DEVOPS ANTI-PATTERNS Dev and Ops Silos DevOps Team Silo Dev Don't Need Ops DevOps as Tools Team Rebranded SysAdmin Ops Embedded in Dev Team web.devopstopologies.com
- 20. 20
- 21. 21 TEAM INTERACTION MODES • Collaboration: 2 teams working together • X-as-a-Service: 1 provides, 1 consumes • Facilitating: 1 team helps another Product teams need a family too - DevOpsDays Portugal
- 22. 22 A BRIEF HISTORY OF ZALANDO TECH
- 24. 24 DEVOPS IS ABOUT SKIN IN THE GAME Absence of skin in the game: People who are isolated from the impacts of their decisions do not learn. They remain captive to their erroneous ideas about how the world works.
- 26. 26 2015: RADICAL AGILITY AWS STUPS DOCKER DEPLOY SSH ACCESS AUDIT REPORTS FULL AWS ACCESS Teams have admin access & full responsibility
- 27. 27 2015: ISOLATED AWS ACCOUNTS Internet *.abc.example.org *.xyz.example.org Team ABC Team XYZ EC2EC2 ELBELB EC2
- 28. 28 Platform > 1100 developers > 200 development teams 2019
- 29. 29 YOU BUILD IT, YOU RUN IT The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. - A Conversation with Werner Vogels, ACM Queue, 2006
- 30. 30 ON-CALL: YOU OWN IT, YOU RUN IT When things are broken, we want people with the best context trying to fix things. - Blake Scrivener, Netflix SRE Manager
- 31. 31 "SKIN IN THE GAME"
- 32. 32 DEVELOPER JOURNEY Consistent story that models all aspects of SW dev
- 34. 34 Developer Journey Correctness Compliance GDPR Security Cost Efficiency 24x7 On Call Governance Resilience Capacity ...
- 35. 35 DEVELOPER PRODUCTIVITY Code Build Test Deploy OperateSetup Cloud Native Application Runtime
- 36. 36 CLOUD NATIVE .. uses an open source software stack to deploy applications as microservices, packaging each part into its own container, and dynamically orchestrating those containers to optimize resource utilization. Cloud native technologies enable software developers to build great products faster. - https://www.cncf.io/
- 37. 37 CONTAINERS END-TO-END Code Build Test Deploy OperateSetup Cloud Native Application Runtime
- 38. 38 CONTAINERS
- 39. 39 CONTAINERS
- 41. 41 PLAN & SETUP
- 42. 42 Plan Stories Rules of Play Tech Radar
- 47. 47 BUILD & TEST
- 50. 50 DEPLOY
- 52. 52 DEPLOYMENT CONFIGURATION ├── deploy/apply │ ├── deployment.yaml │ ├── credentials.yaml # Zalando IAM │ ├── ingress.yaml │ └── service.yaml └── delivery.yaml # Zalando CI/CD
- 53. 53 INGRESS.YAML kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "myapp" servicePort: 80
- 54. 54 TEMPLATING: MUSTACHE kind: Ingress metadata: name: "..." spec: rules: # DNS name your application should be exposed on - host: "{{{APPLICATION}}}.example.org" http: paths: - backend: serviceName: "{{{APPLICATION}}}" servicePort: 80
- 56. 56 CDP: DEPLOY "glorified kubectl apply"
- 59. 59 STACKSET CRD kind: StackSet ... spec: ingress: hosts: ["foo.example.org"] backendPort: 8080 stackLifecycle: scaledownTTLSeconds: 1800 limit: 5 stackTemplate: spec: podTemplate: ... github.com/zalando-incubator/stackset-controller
- 60. 60 TRAFFIC SWITCHING STEPS IN CDP github.com/zalando-incubator/stackset-controller
- 61. 61 EMERGENCY ACCESS SERVICE Emergency access by referencing Incident zkubectl cluster-access request --emergency -i INC REASON Privileged production access via 4-eyes zkubectl cluster-access request REASON zkubectl cluster-access approve USERNAME
- 62. 62 INTEGRATIONS
- 63. 63 CLOUD FORMATION VIA CI/CD ├── deploy/apply │ ├── deployment.yaml # Kubernetes │ ├── cf-iam-role.yaml # AWS IAM Role │ ├── cf-rds.yaml # AWS RDS Database │ ├── kube-ingress.yaml │ ├── kube-secret.yaml │ └── kube-service.yaml └── delivery.yaml # CI/CD config "Infrastructure as Code"
- 64. 64 ZALANDO IAM/OAUTH VIA CRD kind: PlatformCredentialsSet .. spec: application: my-app tokens: read-only: privileges: - com.zalando::foobar.read clients: employee: grant: authorization-code realm: users redirectUri: https://example.org/auth/callback Extensible Kubernetes API
- 65. 65 POSTGRES OPERATOR Application to manage PostgreSQL clusters on Kubernetes >700 clusters running on Kubernetes github.com/zalando/postgres-operator
- 66. Elasticsearch in Kubernetes Elasticsearch 2.500 vCPUs 1 TB RAM github.com/zalando-incubator/es-operator/
- 67. 67 SUMMARY • Application Bootstrapping • Git as source of truth and UI • 4-eyes principle for master/production • Extensible Kubernetes API as primary interface • OAuth/IAM credentials • PostgreSQL • CloudFormation for proprietary AWS services
- 68. 68 Platform as a Product
- 69. “.. means establishing empathy with internal consumers (read: developers) and collaborating with them on the design. Platform product managers establish roadmaps and ensure the platform delivers value to the business and enhances the developer experience.” - ThoughtWorks Technology Radar
- 70. 70 PRODUCT MINDSET ... means we are developing all our products for the best customer experience and biggest business impact while all actions are aligned with our strategy.
- 72. 72 ● Be Prepared. ● Talk to both users and NON users. ● Ask about problems, not solutions. ● Open ended questions. TALK TO USERS
- 73. 73 IDENTIFY COMMON PROBLEMS "Documentation is hard to find" "Documentation is not comprehensive enough" "Remove unnecessary complexity and obstacles." "Get the documentation up to date and prepare use cases" "More and more clear documentation" "More detailed docs, example repos with more complicated deployments."
- 74. 74 TRANSPARENTLY COMMUNICATE WHAT YOU ARE SOLVING
- 76. 76 ENGAGE CUSTOMERS EARLY ON We are working on speeding up builds by providing basic overlays for the most used languages/frameworks so you don't have to download the internet every time you are building your app. This is the approach and the documentation (not live yet), please add your comments to guide us in the right direction.
- 77. 77 SUPPORT IS A FEATURE
- 78. 78 DOCUMENTATION • Restructure following www.divio.com/en/blog/documentation/ • Concepts • How Tos • Tutorials • Reference • Global Search • Weekly Health Check: Support → Documentation
- 80. 80 COMMUNICATION Guidelines • Target announcements • Use the right tags • Timing
- 81. 81 NEWSLETTER "You can now.." • You can now upload your Scala code coverage report to SonarQube "This enables you to.." • This enables you to browse your code quality results and track it over time
- 82. 82 “Useful information, good level of details and pleasant to read. It's one of the few newsletters that I took time to read entirely :)” - a reader, July 2018
- 84. 84 WITH GREATER POWER COMES GREATER RESPONSIBILITY
- 85. 85 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate srcco.de/posts/accelerate-software-delivery-performance.html
- 86. 86 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate • Commit to Prod • Deploys/week/dev • MTRS from incidents • N/A
- 87. 87 DELIVERY PERFORMANCE METRICS • Lead Time • Release Frequency • Time to Restore Service • Change Fail Rate • Commit to Prod • Deploys/week/dev • MTRS from incidents • N/A
- 90. 90 INSIGHTS THAT GUIDE THE PRODUCT “Kubernetes/CDP deploys are too slow - ours takes over 10 minutes, mainly due to having to download SBT dependencies every time.”
- 91. 91 INSIGHTS THAT GUIDE THE PRODUCT “CDP should somehow cache builds or something like that. The current approach forces everyone to re-invent the cache wheel for their own projects - or face a crazy slow build.”
- 92. 92 INSIGHTS THAT GUIDE THE PRODUCT “Kubernetes/CDP deploys are too slow - ours takes over 10 minutes, mainly due to having to download SBT dependencies every time.”
- 93. 93
- 94. 94 TESTIMONIALS “So, thank you, Team Automata, for listening to our community, taking our upvotes in consideration when developing new solutions and building every day 'the first CI that doesn't suck'.” - a user, October 2018
- 95. 95 TESTIMONIALS “Thank you very much for improving the color scheme of the UI. It really helps me with my color blindness. It used to be hard to distinguish between statuses previously.” - a user, February 2019
- 96. 96
- 97. 97 Wrap Up
- 98. 98 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 99. 99 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 100. 100 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 101. 101 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 102. 102 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 103. 103 DEVOPS IS … if every person uses the same tool for the same job … codified knowledge - everybody contributing their part to common automation … if all people have the same privileges in their tooling … if human error is equally possible for Dev and Ops … replacing people interfaces by automated decisions and processes … a result bit.ly/5pdops
- 104. 104 DevOps is normal
- 105. 105 DevOps is normal
- 106. 106 You build it, you run it is normal
- 107. 107 WAIT A SECOND.. What about my job title?
- 108. 108 "DEVOPS ENGINEER" AS A JOB TITLE
- 109. 109 T-SHAPED ENGINEER "Generalizing specialist" Breadth of Knowledge TechnicalDepth
- 110. 110 DevOps Engineer Software Engineer / Software Developer (T-Shaped)
- 111. 111 ON THE DEVOPS JOB TITLE "Everyone’s a devops engineer these days. I’m a devops engineer, my wife’s a devops engineer, even my dog’s a devops engineer" - James Betteley, 2014 devopsnet.com/2014/08/13/the-devops-team-myth/
- 112. 112
- 113. 113 OPEN SOURCE & MORE Kubernetes on AWS github.com/zalando-incubator/kubernetes-on-aws Skipper HTTP Router & Ingress controller github.com/zalando/skipper External DNS github.com/kubernetes-incubator/external-dns Postgres Operator github.com/zalando-incubator/postgres-operator More Zalando Tech Talks github.com/zalando/public-presentations