Curiosity Software Presents: Modelling for Continuous Testing
Download as PPTX, PDF0 likes47 views
The document discusses challenges in current testing approaches within DevOps, highlighting issues like heavy reliance on UI-based end-to-end (e2e) tests, inadequate test coverage, and delays due to complicated test environments. It advocates for a shift towards a more efficient testing model that emphasizes automation, collaboration, and continuous feedback to enhance testing practices and reduce defects. The framework presented emphasizes principles of DevOps, such as flow, amplifying feedback loops, and fostering a culture of continual experimentation and learning.
Curiosity Software Presents: Modelling for Continuous Testing
- 1. © Curiosity Software Ireland Ltd. 2023 Modelling for Continuous Testing February 28th 2023 Ben Riley CIO, Curiosity Ben.Riley@Curiosity.Software @CuriositySoft Rich Jordan Enterprise Solutions Architect, Curiosity Rich.Jordan@Curiosity.Software @RichJordan81
- 2. © Curiosity Software 2023 You have a DevOps Transformation, but your Test approach hasn’t really changed • You have a large team of testers doing mainly e2e tests (even if they call it something else). • You have a test pack largely made up of UI based e2e tests which always grows after each release – you suspect there’s lots of duplicated effort in there. • Testing is delayed because you are waiting on large, complicated e2e environments to be built, or waiting for your slot in the few e2e environments that persist in your organization. • Test coverage relies on traceability back to requirements that you know are incomplete or out of date. • Test design is largely an activity that involves testers creating test cases without input from the wider development team. Alternatively, you don’t design test at all; you just write test cases/automation. • A large % of your defects are environment/config related, causing large delays in test phases. • You are finding fundamental design issues in late, large, slow e2e test phases. • Despite trying many different tools, you are struggling to increase the rate of tests executed through automation as the tests are just too flaky! Original Image: Khanargy, Wikimedia Commons, published under CC BY-SA 4.0 license.
- 3. © Curiosity Software 2023 Your Test approach and the Testing Pyramid • Traditional UI testing is too slow • Doesn’t lend itself to stable, repeatable automation at scale • Quickly requires lots of real e2e environments • Doesn’t get to the real route cause of problems
- 4. © Curiosity Software 2023 DevOps: what are the 3 ways of DevOps Principle The First Way: Flow/Systems Thinking. The Second Way: Amplify Feedback Loops. The Third Way: Culture of Continual Experimentation and Learning. The Three Ways: The Principles Underpinning DevOps by Gene Kim / IT Revolution https://itrevolution.com/articles/the-three-ways-principles- underpinning-devops/ Original Image: Khanargy, Wikimedia Commons, published under CC BY-SA 4.0 license.
- 5. © Curiosity Software 2023 Flow • Never passing a known defect to downstream work centers • Never allowing local optimization to create global degradation • Always seeking to increase flow • Always seeking to achieve profound understanding of the system What do testers actually spend their time doing? Typical Test team effort Comprehension of system / change 30% (Automation) Test script creation/maintenance 20% Test execution 20% Waiting to test 5% Evidencing / Reporting 10% Other (Meetings / Self- Development) 15% 100% Sample represents 50 testers Software entropy is the idea that software eventually rots as it is changed if sufficient care is not taken to maintain coherence with product design and established design principles. Managing Software entropy 1.Improve the communication in your team. 2.Maintain coherence with product design and established design principles
- 6. © Curiosity Software 2023 Flow • Never passing a known defect to downstream work centers • Never allowing local optimization to create global degradation • Always seeking to increase flow • Always seeking to achieve profound understanding of the system Diagram adapted from John Ferguson Smart’s “BDD at the heart of any DevOps Transformation” 3) We relentlessly automate testing and checking of the models in test execution! If we don’t know the outcomes, we find out through the process of building the models… we ask critical questions and do critical thinking 1), 2) We build a common understanding of the stack and how its changing through collaborative teams, building models with engineered, predictable & repeatable outcomes! 4) We execute all checks as part of the standard CI/CD process! 4a) We are driven by the metric we gather to provide quality and lean improvements wherever they exist 5) We play an active role in promoting best practices and celebrating continual improvements Illustrate Formulae Automate Validate Demonstrate Feedback Visibility Valuable Software Testers Business Rules Examples A Shared Understanding Executable Specifications, Wireframes… Automated Acceptance Tests Living Documentation 1 2 3 4 5 BA Business Developers
- 7. © Curiosity Software 2023 Amplify Feedback Loops • Shortening and amplifying all feedback loops • Embedding knowledge where we need it. • Shifting Test left in engaging design – robust test design of rules and flows – we build models • Quick to build, stable and quick to run automated tests • Fail fast and fail early – once built, API test runs in line with DevOps cadence for delivery orchestrated inline with build deployment • Iterative building block approach enables Continuous Integration delivery Manual Tests Integration Tests Unit Tests E2E Tests (UI Testing) What the automation pyramid says good looks like UI / UX / Look and Feel /GUI Infrastructure Device Client / Functionality / API Interface / SOA Rules / Services System of Record / ETL Data Transition Rules / Batch Services Comparison Rules Engines E2E Acceptance UI Channel Change Going Beyond The Test Pyramid (everything is a model) App / Client Functional Change / Contracts Platform / Infra Build Interface Change / Contracts ETL Validation / Big Data Stubs mapped against understood coverage and version- controlled interface spec Stubs mapped against understood coverage and version- controlled interface spec
- 8. © Curiosity Software 2023 Shortening and amplifying feedback loops – modelling for isolation 1) Reinforcing / proving the external boundaries 2) Reinforcing / proving the internal boundaries 3) Proving the systems elements in isolation 4) Proving batch / reporting 5) Proving the golden thread of the platform 6) Stakeholder demonstration The more this last stage deemed to be testing, the more monolithic the platform and change will become
- 9. © Curiosity Software 2023 Culture of Continual Experimentation and Learning. • Continual experimentation • Taking risks and learning from failure • Understanding that repetition and practice is the prerequisite to mastery Models help us to understand our systems & change impact Refactor and optimize based on isolated risk Minimized the blast radius (the impact of failure)
- 10. Demo © Curiosity Software Ireland Ltd. 2023
- 11. © Curiosity Software 2023 Flow • Never passing a known defect to downstream work centers • Never allowing local optimization to create global degradation • Always seeking to increase flow • Always seeking to achieve profound understanding of the system Diagram adapted from John Ferguson Smart’s “BDD at the heart of any DevOps Transformation” 3) We relentlessly automate testing and checking of the models in test execution! If we don’t know the outcomes, we find out through the process of building the models… we ask critical questions and do critical thinking 1), 2) We build a common understanding of the stack and how its changing through collaborative teams, building models with engineered, predictable & repeatable outcomes! 4) We execute all checks as part of the standard CI/CD process! 4a) We are driven by the metric we gather to provide quality and lean improvements wherever they exist 5) We play an active role in promoting best practices and celebrating continual improvements Illustrate Formulae Automate Validate Demonstrate Feedback Visibility Valuable Software BA Testers Business Developers Business Rules Examples A Shared Understanding Executable Specifications, Wireframes… Automated Acceptance Tests Living Documentation 1 2 3 4 5
- 12. © Curiosity Software Ireland Ltd. 2023 Questions and Discussion
- 13. © Curiosity Software 2023 Carry on the conversation? • Book a demo and discussion with a Curiosity expert! • Try Test Modeller. Visit Testmodeller.io for a free trial and online tutorials. • Follow Curiosity Software on LinkedIn and @CuriositySoft on Twitter for resources and news. • Connect with Ben and Rich on LinkedIn.
- 14. © Curiosity Software Ireland Ltd. 2023 Thank you Ben.Riley@Curiosity.Software Rich.Jordan@Curiosity.Software
Editor's Notes
- #2: And onto our topic for today, “Modelling for Continuous Testing”. It’s my pleasure to hand over to Rich and Ben, to introduce themselves and get us started today.
- #12: Test creation remained manual Scripting is too repetitive – even with boilerplate code and page objects Manual test case design cannot overcome system complexity risky over-testing and wasteful over-testing Test data breaks automation Brittle tests create impossible maintenance Automation becomes a “black box” and silo
- #13: Thank you Rich and Ben for the fantastic presentation! We’ll now open for questions, so please do get your questions to Rich and Ben in the Q&A box. Why don’t we start with this question: Where do I start We have lots of existing tests. Do we need to throw them away? Does this work against rest assured?
- #14: Fantastic, thank you everyone for your questions and for joining today! If you’d like to carry on your learning today, there are a range of ways you can do this. All these links will be shared with you in a follow-up email, and we’ll also share the slides. Firstly, if you’d like to talk to us, please book a meeting with a Curiosit expert. If you’d like to get hands on, you can also sign up for a Test Modeller trial, using our range of online tutorials to get started. And please do follow Curiosity on LinkedIn and Twitter to keep up with all the latest updates. You can also find Ben and Rich on LinkedIn, if you’d like to connect with them both there.
- #15: So, we’re at time – and all I think remains to be said is a huge thank you to everyone who’s joined today, and an even bigger thanks to Ben and Rich for presenting!