Building Quality In in SAFe – The Testing Organization’s Perspective
2 likes1,362 views
The document outlines strategies for improving quality in agile testing organizations, emphasizing the importance of early defect detection and continuous integration. It presents a framework for assessing current testing practices and encourages the adoption of Agile Testing principles, such as Acceptance Test-Driven Development (ATDD) to enhance collaboration between developers and testers. It advocates for shifting testing left in the development cycle, increasing automation, and fostering a culture of shared quality responsibility within agile teams.
![“We used to feel
comfortable to
release quarterly and
anxious to release
every two weeks. “ “Now releasing every
two weeks is natural. We
are anxious when we
can’t do it [due to holiday
freeze]”
FiftyOne.com](https://image.slidesharecdn.com/nnphw2kiqcevc9ugrju4-signature-babdd967c702ec4fdde6612b35192d435caddf382cb68a0b72e7778378d062b2-poli-181004211604/85/Building-Quality-In-in-SAFe-The-Testing-Organization-s-Perspective-4-320.jpg)
Building Quality In in SAFe – The Testing Organization’s Perspective
- 1. 1© Scaled Agile, Inc. Building Quality In The Testing Organization’s Perspective
- 2. 2© Scaled Agile, Inc.© Scaled Agile, Inc. 2© Scaled Agile, Inc. Yuval Yeret SPCT, CTO AgileSparks, Inc
- 3. Limit Defects in Process – Early Quality is Cheap Quality
- 4. “We used to feel comfortable to release quarterly and anxious to release every two weeks. “ “Now releasing every two weeks is natural. We are anxious when we can’t do it [due to holiday freeze]” FiftyOne.com
- 5. But how does the typical Testing organization leader feel when seeing the Continuous Delivery Pipeline vision?
- 6. See Kent Beck’s idea as described by Markus Gartner at http://www.shino.de/2010/11/04/software-g-forces-the-effects-of-acceleration/
- 7. Agile Team2 Agile Team1 Ongoing Done Features Backlog Develop Feature/ Sprint in Progress Story-level Test & Fix Deploy ment Done Ongoing Done Stories Backlog End of Release Testing Ongoing DoneOngoing Specify Done D E F A B C G H J K L M H.6 H.0 H.1H.2 H.3 H.4H.5 D2 D3 T2 T1 D1 P1 H.7 T2 Implement Feature by Stories UAT Regression Performance Security Functional Progression Exploratory ATDD Aut o Long wait for the endgame In real life – Agile Testing at the Story level, Waterfall Testing at the Release level. Continuous Integration a tough challenge - Continuous Delivery a faraway dream “Iteration is too short for everything we need to achieve DONE” “Let’s leave the serious testing for the IP iteration” The 2-level Test Strategy Pyramid – Story + Release-level Platform Matrix Real Network
- 8. The result - only a limited amount of feedback is early and effective • A major amount of issues are discovered only in the “end game” • finding defects late increases the cost to fix them exponentially • The result – the need to do a long “end game” to accommodate fixing them, retesting, and even that turns out not to be enough in some cases when the complexity of the big bang “end game” is overwhelming. 0 20 40 60 80 100 120 1 3 5 7 9 11 13 15 17 19 21 23 done done done60%4 0 % 40% 60% Time to Detect
- 9. Self-Assess – Continuous Testing Dimension • Individually assess where you are: 1. Sit – All testing happens towards end of release. 2. Crawl – Some testing happens by Testing team in follow up sprints. Most still towards end of release. 3. Walk – Majority testing happens inside the Agile teams. Considerable amount still happens before release. 4. Run – Minimal deployment/release-testing is needed – IP iteration isn’t heavily used for Testing/Hardening 5. Fly – All testing is run continuously. System is always ready for deployment/release. 9
- 10. #1 - Take an economic view #2 - Apply systems thinking #3 - Assume variability; preserve options #4 - Build incrementally with fast, integrated learning cycles #5 - Base milestones on objective evaluation of working systems #6 - Visualize and limit WIP, reduce batch sizes, and manage queue lengths #7 - Apply cadence, synchronize with cross-domain planning #8 - Unlock the intrinsic motivation of knowledge workers #9 - Decentralize decision-making SAFe Lean-Agile Principles Which SAFe Lean/Agile Principles are we struggling with here?
- 11. Agile Team2 Agile Team1 Ongoing Done PI Backlog Build Feature/ Sprint in Progress Story-level Test & Fix Deploy ment Done Ongoing Done Team Backlog Ready For Feature Test Test Feature- level Ongoing Done End of Release Testing Ongoing DoneOngoing Define Done D E T F T T A B C G H J K L M H.6 H.0 H.1H.2 H.3 H.4H.5 D2 D3 T2 T1 D1 P1 H.7 T2 Implementation UAT Regression Performance Security Functional Progression Exploratory First step forward - Achieve Agile Testing Flow at the Story AND Feature/Epic/Iteration level Add Feature/Epic level testing 1. Add Feature/Iteration level testing. Avoid the long wait while minimizing costs 2. left-shift more and more testing through automation, enabling teams using environments/tools/knowhow and more. The 3-level Test Strategy Pyramid - Story, Feature/ Iteration, Release Tools, Simulators and environments available to developers during development enabling them to build the software in consideration of the real world environment
- 12. Optimize Testing Batch Sizes • For each testing type: – Create the economic batch size tradeoff curve – Figure out Ideal batch size without further investment – Identify enablers that would enable shifting batch size to the left and improving overall economics 12 Learn more - http://docs.perfectomobile.com/docs/resources/paper /digital-quality-in-build-cycle.pdf
- 13. The Test Automation Pyramid • Most expensive automation to develop, run & maintain, so minimize!!! • Move majority of E2E testing coverage to Service/API layer • QTP/UFT/Selenium/PerfectoMobile/etc. UI • “The Workhorse” of enterprise agile testing • Created by testers & developers on agile teams supported by frameworks/guidance by Automation CoE • soapUI, etc. Acceptance (Service/API) • Leverage Agile Teams developer testing to reduce coverage needs • Ability to automatically detect (through coverage tools etc.) what is covered Unit Testing Manual http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid
- 14. Self-Assess – Test Automation Pyramid • Individually assess where you are: 1. Sit – Most regression testing is actually manual. There’s some UI based coverage but can’t keep up. 2. Crawl – Majority of the regression test suite is based on UI-level automation. It is hard to maintain, slow-running, and brittle – often fails without reason and requires lengthy analysis. (Semi-Automated) 3. Walk – Inverted pyramid – Developers are writing unit tests for new code that reduces some need for other coverage for that code, Some API testing exists, Still a lot of UI based coverage. Some manual testing is needed beyond the automation pyramid. 4. Run – Tube – Similar amount of coverage coming from each area. Minimal manual testing still needed. 5. Fly – Upside pyramid with no manual regression testing required and fully automatic and reliable continuous testing integrated into continuous integration. 14
- 15. 15
- 16. In classic testing – we find defects to assure quality Elaborate Requirements Technical Design Coding / Unit Testing Test Execution + Fixing Done Test Design Classic agile development – Test Design in parallel or after coding Test automation - Happens towards the end of the agile iteration or is left over for next iterations / automation team. Test Automation
- 17. Shift even more to the left… – Specify Acceptance Tests FIRST to DRIVE Design/Development Elaborate Requirements Test Design Technical Design Coding / Unit Testing Test Execution + Fixing Done 1. ATDD Thinking Use test scenarios to guide design ATDD = Acceptance Test Driven Development - Build Quality Into Design – preventing defects rather than just finding them (a.k.a BDD – Behavior Driven Development) http://www.agilesparks.com/test-first-reading-list Test Automation
- 18. Agile Team2 Agile Team1 Done Ongoing DoneOngoing Done D E T F T T A B C G H J K L M H.6 H.0 H.1 H.2 H.3 H.4 H.5 D2 D3 T2 T1 H.7 T2 Specify/Groom using Acceptance Tests Backlog Grooming / Acceptance-Tests Specification Workshop • Identify acceptance criteria/tests for the next stories in the backlog. • Use acceptance tests as guidance for slicing stories smaller to enable more effective agile collaboration Ongoing Done PI Backlog Build Feature/ Sprint in Progress Story-level Test & Fix Deploy ment Done Ongoing Done Team Backlog Ready For Feature Test Test Feature- level Ongoing Done Release Testing Ongoing DoneOngoing Define Done Implementation
- 19. Acceptance Tests/Mini- Stories StoriesFeatures Epic Feature H Story H1 Test for H1 Mini-story H4 Mini-story H5 Story H2 Story H3 Feature I Feature J Feature K
- 20. Agile Team3 Agile Team2 Agile Team1 D Done Done A B C D F F E E K H Z J A.1 A.2 A.3A.4 A.5 D D T T Team pulls a few stories Testers create more detailed test examples Developers use them to guide technical design/specification Ongoing Done PI Backlog Build Feature/ Sprint in Progress Story-level Test & Fix Deploy ment Done Team Backlog Ready For Feature Test Test Feature- level Release Testing Define Implementation OngoingOngoing Done Ongoing Done Ongoing Done Ongoing OngoingOngoing Done
- 21. Agile Team3 Agile Team2 Agile Team1 D A B C D F F E E K H Z J A.1 A.2 A.3A.4 A.5 D D T T Story is pulled into development Testers focus on detailed test plans, automation preparation, data preparation. Developers code and do developer/unit/small testing DoneOngoing Done PI Backlog Build Feature/ Sprint in Progress Story-level Test & Fix Deploy ment Done Team Backlog Ready For Feature Test Test Feature- level Release Testing Define Implementation OngoingOngoing Done Ongoing Done Ongoing Done Ongoing OngoingOngoing Done
- 22. Agile Team3 Agile Team2 Agile Team1 OngoingOngoing Done A B C D F F E E K H Z J A.1 A.2 A.3 A.4 A.5 D D T T D Developer verifies the story passes the team “ready for story testing” criteria: • Code is “in the build” • The acceptance tests for this story pass in the dev env • Good code coverage using automated unit tests • Sanity test passes Developer marks story as dev done, ready for testing Development done means the acceptance tests already pass Ongoing PI Backlog Build Feature/ Sprint in Progress Story-level Test & Fix Done Team Backlog Ready For Feature Test Test Feature- levelDefine Implementation Ongoing Done Done Ongoing Result: Teams using ATDD get much higher, automated, test coverage(some above 90%) than the typical test- last teams. Both due to discipline as well as due to developers awareness to the acceptance tests
- 23. Agile Team3 Agile Team2 Agile Team1 A B C D F F E E K H Z J A.1 A.2 A.3 A.4 A.5 D D T T D Tester pulls story into testing • Tester finds significantly fewer defects • Product Owner is happier with how the story works & finds fewer surprises. • Both as a result of deeper alignment achieved in the specification workshops with ATDD scenarios. Developers meanwhile pull more stories into coding DoneOngoing Done PI Backlog Build Feature/ Sprint in Progress Story-level Test & Fix Deploy ment Done Team Backlog Ready For Feature Test Test Feature- level Release Testing Define Implementation OngoingOngoing Done Ongoing Done Ongoing Done Ongoing OngoingOngoing Done
- 24. Self-Assess – Test-First Dimension • Individually assess where you are: 1. Sit – Test design, execution, and automation happens after software is designed, when interfaces are stable. Story/Iteration Definition of Done doesn’t include testing 2. Crawl – Test design and execution is included in definition of Done and executed by the Agile team – test design for each story happens after the story is build/coded. 3. Walk – Test design happens in parallel to defining/building the story. Feedback from test design can sometimes influence software functionality/design choices. 4. Run – Test design happens FIRST and is always used as feedback/guidance when defining the story. 5. Fly – Test design happens FIRST and is used to create mini-stories each focusing on one small slice/scenario that can then be designed and built and tested very quickly by the team enabling fast effective flow 24
- 25. 25
- 26. All together now– Whole Team Automation guided by ATDD thinking and the Test Automation Pyramid UI Acceptance (Service/API) Unit Testing Elaborate Requirements Test Design Technical Design Coding / Unit Testing Test Execution + Fixing Done 1. ATDD Thinking Use test scenarios to guide design 2. ATDD Automation Using BDD tool and the relevant dev- friendly test tools Built in parallel to development at the right level (according to the test automation pyramid)
- 27. ATDD/BDD Tools SoapUI Selenium QTP Other… ATDD/BDD Tools Options and Architecture Andothers… Act as a layer on top of test drivers that actually “drive” the SUT (System Under TesT) Web Services Web Browser Web Browser + Fat Clients Gherkin Language Table-Driven Mobile
- 28. Self-Assess – Test-First Tooling Dimension • Individually assess where you are: 1. Sit – relying on record&play techniques and cannot support test-first. 2. Crawl – Developers can automate during/before development using APIs. 3. Walk – Whole team can automate significant amounts of coverage during/before development using a combination of APIs to the system under test and an ATDD/BDD tool enabling non-developers to create scenarios. 4. Run – ATDD/BDD tooling runs all functional testing, is integrated into CI, and is also used for some of the non-functional tests. 5. Fly – We’re super awesome at this, we do it all the time, we’re not willing to ever go back! 28
- 30. Testers need to provide a different kind of value within an agile team/organization • Being Champions of the Product and the Customer/User. • Specializing in Performance/ Security/Load/etc. • Shining light on where to focus quality efforts by analyzing risk probability and Impact.
- 31. • QA - Accountable to Quality: By Enabling it rather than Owning it • Focus on elaborating scenarios to help guide the team as well as advanced and exploratory tests – especially for the riskier stories • Automation development and execution – a team responsibility – Developers typically handle the majority of the automation development • ATDD/BDD tools enable QA people to guide the team and enable automation without being coders themselves. Agile Tester as Test/QA expert rather than automation engineer
- 32. Ongoing Done Develop Feature/ Sprint in Progress Story-level Test & Fix Deploy ment Done Ongoing Done Stories Backlog Ready For Feature Test Test Feature -level Ongoing Done End of Release Testing Ongoing DoneOngoing Specify Done Features Backlog Most Agile Testers join the Agile teams to help “Build Quality In”. In some cases the System Team takes on deeper more specialized testing tiers – like full coverage of the compatibility matrix A B C D E F H J GKL M I Agile Team Test Focal Business/Do main focused (onshore) Agile Testers Tech. focused Auto./Man ual (possibly near/offsho re) System Team Agile Team2 Agile Team1 D2 D3 T2 T1 D1 T2 Test Manage r Testing CoP T T T T Manage test strategy, capabilities, Retrospect, Mentor/Coach Spread testers in Agile teams and the System team according to your Test Strategy Pyramid
- 33. Self-Assess – The Agile Tester Dimension • Individually assess where you are: 1. Sit – Testers are in a different organization on different teams. 2. Crawl – Teams get allocated ”half a tester” 3. Walk – Agile Teams include testers that focus on that team and can really support Test-First efforts. 4. Run – Agile Testers cover the majority of testing work including some of the more specialized test types 5. Fly – Teams look up to their testers to guide the way towards quality – test-first isn’t just a process it is the way the teams think and behave. 33
- 34. 34
- 35. Summary - Navigate the SAFe Implementation Roadmap from the Testing Organization’s perspective Testing/QA Leader for this ART Testing/QA Leader for the organization Testing/QA Leader for the organization Figure out impact on Dev/Test Infra, Quality Practices, Create System Team Testing/QA Leader for the organization Product Ownership for Test Automation Backlog of Enablers All Testers/QA Engineers participate Test-First Automation + Allocate capacity to close automation gaps Champion building quality in! Inspect and Adapt organizational Quality metrics Shift more and more testing left, Reduce batch sizes, bring more and more capabilities into the Agile teams
- 36. 36© Scaled Agile, Inc.© Scaled Agile, Inc. 36© Scaled Agile, Inc. Questions
- 37. 37© Scaled Agile, Inc.© Scaled Agile, Inc. 37© Scaled Agile, Inc. Thank you! Coming soon - presentation downloads at: safesummit.com/presentations
- 38. 38© Scaled Agile, Inc.© Scaled Agile, Inc. 38© Scaled Agile, Inc. Please rate this session Open “Schedule” on mobile app and locate session Tap star rating at top of screen