What is quality, and how do we build it in
- 1. What is quality, and how do we build it in? @maryamumar | @jezhumble | #DeliveryConf
- 2. Agenda ● Introduce Thought Machine ● How to get started with improving quality ● Selecting metrics ● Evolving metrics based on what we learn @jezhumble @maryamumar
- 3. TLDR ● Quality is subjective - what to focus on depends on many factors ● Start by creating a vision, selecting some metrics, and measuring ● What you measure will evolve, and will determine what teams work on @jezhumble @maryamumar
- 4. Please fix Quality! Clients are unhappy Issues are found late Releases are delayed Team feels stressed
- 6. Common mistakes we have observed ● Stop work and build a comprehensive automated test suite ● Add new process and approvals to try and prevent escaped defects ● Create a new “quality” function which sits downstream of development ● Underinvestment in infrastructure and tools @jezhumble @maryamumar
- 7. Create a Plan But where do you start?* *Leading Quality - Ronald Cummings-John, Owais Peer
- 8. Vision Statement As an <engineering team/customer> When <there are any changes to production code> Then <every merge to the codebase is releasable> And <there are no known regressions in the system> @maryamumar
- 9. Assuring Quality enables teams to drive for customer satisfaction at a sustainable pace.
- 10. Who are your Stakeholders? ● CEO/CTO ● Tech leads ● Testing team ● Incident Management ● Release Management Questions to ask: ● How do you know quality was bad? ● Escaped defects? ● What’s your code coverage like? ● Do you have the right processes in place? @maryamumar
- 11. Software Delivery Performance @jezhumble https://cloud.google.com/devops/state-of-devops/
- 12. How to get better https://bit.ly/dora-bfd@jezhumble
- 13. Areas to Improve Releases Product Quality Process Quality @maryamumar
- 14. Releases - Create visibility landscape for releases - Monitor issues found post-release vs pre-release - How good is your regression cycle? - Release notes - Changes - Known issues; when to expect fixes - How easy is it to create deployables? - How quickly can you deploy? @jezhumble @maryamumar
- 16. Product Quality @jezhumble @maryamumar MTTG Build FailuresBuild Time Deploy Time Average Time in Status PR Review PR commit rate per Sprint %age of Flaky Tests Ticket Fluctuation from QA to Dev Test Coverage
- 17. Product Quality (contd.) @jezhumble @maryamumar Defects found in Sprint vs Escaped Defects Team FeedbackDefects found via Automation via Exploratory testing per Feature Bug resolution time per Severity level
- 20. Time from check-in to deployed in test environment
- 21. SATs vs Defects per Team
- 22. High trust culture Westrum, “A Typology of Organizational Cultures” | http://bmj.co/1BRGh5q how organizations process information
- 23. Psychological safety “Wherever there is fear, you will get wrong figures” — W Edwards Deming @jezhumble @maryamumar
- 24. https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/ Culture of psychological safety ● Predicts software delivery performance and organizational performance ● Implement by adopting continuous delivery and lean product management practices @jezhumble
- 25. 2018 State of DevOps Report: cloud.google.com/devops SDO Performance Trust Software Delivery Performance Availability Organizational Performance Voice Westrum organizational culture Autonomy Retrospectives Climate for Learning @jezhumble
- 26. How to make metrics safe ● Teams involved in creating vision ● Consult with teams when deciding what to measure ● Testing team makes quality visible ● Let teams set their own targets ● Regular retrospectives on what is being measured @jezhumble
- 27. Conclusion ● Quality is subjective - what to focus on depends on many factors ● Start by creating a vision, selecting some metrics, and measuring ● What you measure will evolve, and will determine what teams work on © 2020 Google LLC & Thought Machine Group Ltd
- 28. Parking lot
- 29. Continuous testing ● Continuously reviewing and improving test suites to better find defects and keep complexity and cost under control ● Allowing testers to work alongside developers throughout the software development and delivery process ● Performing manual test activities such as exploratory testing, usability testing, and acceptance testing throughout the delivery process ● Having developers practice test-driven development by writing unit tests before writing production code for all changes to the codebase ● Being able to get feedback from automated tests in less than ten minutes both on local workstations and from a CI server 2018 State of DevOps Report | cloud.google.com/devops
- 30. Continuous delivery The ability to get changes—features, configuration changes, bug fixes, experiments—into production or into the hands of users safely and quickly in a sustainable way. 2018 State of DevOps Report | cloud.google.com/devops