Agile San Diego: Testing as Exploration (Continuous Delivery w/o Automation)
Download as PPTX, PDF0 likes955 views
This document discusses using exploratory testing to enable continuous delivery without test automation. It describes how one company shifted from a Scrum process with late, poorly tested monthly releases to a Kanban approach using exploratory testing across feature branches. This allowed the company to continuously deploy features to production as soon as developers determined they were tested and releasable, significantly reducing lead times. Key enablers included dropping estimates, limiting work-in-progress, pairing testers and developers, and using exploratory testing to focus on actionable feedback over predefined test cases. The changes improved collaboration and allowed the team to progressively improve testing practices over time.
Agile San Diego: Testing as Exploration (Continuous Delivery w/o Automation)
- 1. Testing as Exploration aka. Continuous Delivery without Automation Maaret Pyhäjärvi Email: <maaret@iki.fi> | Twitter: maaretp Maaret Pyhäjärvi Nimeä | Attribution (Finland) http://creativecommons.org/licenses/by/1.0/fi/ http://creativecommons.org/licenses/by/1.0/fi/deed.en
- 2. 1. LET’S TALK ABOUT EXPLORATORY TESTING ?
- 3. Things Can Look Different from Different Perspectives
- 4. Realizations about Nature of Testing 20 16 1639 5±2 4
- 5. Exploratory Testing: Better tests, better testers! • An approach, not a technique • Find unknown unknowns • Disciplined • Test is a performance, not artifact – Artifacts support human memory – Many forms: e.g. checklists and automation • Exploratory performance testing, Exploratory test automation, Exploratory regression testing Test-related learning Design of new tests Test execution Result interpretation 5
- 6. Exploratory Testing: Learning & Modeling ”A day’s work” Vision (“Sandbox”) Current Charter Other Charters Details Bug Reports Perception of quality and coverage Quality ReportDebriefing Tester Test Manager Past Results Obstacles Outlook Feelings ? # xCharter backlog of the future testing Out of budget Next in importance! #, ?, x, + 20:20:60 Session sheets of the past testing Idea of exploration Metrics summary Coachin g 6 Playbooks Coverage outlines
- 7. 2. CONTINUOUS DELIVERY WITHOUT AUTOMATION
- 8. Where does this say fast? Where does this say automation ?
- 9. A Bit of Background • 3 years into a web based (.NET/C#) product • 1:8 tester:developer ratio • Missing agile technical practices: test automation non-existent • Scrum-like monthly releases for first 2 years – Releases late – Releases not tested well enough • Negative cycle of failing with estimates eating team morale • Jira in a central role with estimates and burndown
- 10. Going for Continuous Delivery
- 11. The Main Driver for Change: Testing Scheduled release Feature 1 Feature 2 Feature 3 Feature 4 Testing Feature 1 Feature 2 Feature 3 Feature 4 R1 R2 R3 Done includes Tested Schedule over Quality SCRUM + SCHEDULED DELIVERY with continuous integration KANBAN + CONTINUOUS DELIVERY
- 12. Enablers that Made Timing Just Right • Henri Karhatsu and #NoEstimates – experience report • Availability of Git in Visual Studio without plugins • Problems with schedules in Scrum • Lean and value delivery focus in Product Management
- 13. Changing How We Managed Our Work: Setting up Jira Kanban Removed hour & story point estimates Agreed in WIP for each phase Agreed on swarming Updated issue types
- 14. Branches and Test Environments Feature Test Environment Feature branches on-demand builds “Developers think this can be tested” Development Test Environment Integration branch build-on-checkin “Developers think this can be released” Acceptance Test Environment Master branch with fixes and merges from integration on-demand builds “Release next morning”
- 15. Continuous Deployment but Significant Lead Times Mon Tue Wed Thu Fri Mon Tue Wed Thu Fri Change to Master Integration to master Feature to integration Developers drive the decision on what they want feedback on
- 16. Exploratory Testing Enables Continuous Deployment • Every Jira task gets planned for – Sometimes we go with developer testing only – Sometimes we test extensively • Exploratory Planning – No set test cases – Talking to developers and reflecting to a model of of use created through earlier explorations – Actionable information first –principle – Production monitoring is an option for getting information
- 17. Changes • Active discussion about schedules and merging, and needs of testing in the branches • More pairing for testing the features • More group work on defining the features • Introducing Flowdock due to increased need to collaborate; integrating logs, emails • Focus on getting better (scope of test automation; refactoring; pairing and group work; individuals’ skills)
- 18. Testers don’t break your code, they break your illusions about the code. -- adapted from James Bach
Editor's Notes
- #3: WHY we test: for information. To avoid the problems in production. Tekes case. Someone explored and got classified data. Instead of a thank you and reward, they got arrested. Use you skills in a test environment. Example: Security vulnerabilities Vulnerabilities are considered real problems Still only the black market pays real money for reporting them Functional bugs / missing value reporting is actual effort too, and wastes time for people Something we as end users can do: reclamation
- #4: We want it to work in production. And with developer-tester perspectives, we’ve managed to make that true. We had recruited a tester because we knew (the devs) that they couldn’t see things from all the necessary angles. So they had asked for the feedback.
- #5: Lisähuomiona: perinteiset testausvaiheet väärin nimetty, ovat korjausvaiheita. Hyväksymistestaus on ainoa testausvaihe – ja vain mikäli järjestelmätestaus on tuotannonkaltaista. Boltonin sertifiointi: "I test. You observe. We talk. You decide".
- #7: Remember to talk about tacit knowledge and the fact that there’s skills and heuristics that are teachable that testers tend to specialize on…
- #9: Fast as in few hours. Fast is embedded in responding to change: some change comes only from testing the value in real production environment. Just saying we can take best of the context we have and start living to the values.
- #13: Value in production
- #16: With 9 developers, it’s possible to end up having a feature per day ready for production…
- #17: Happy agile team with satisfied customers with collaboration and smart exploratory testing, not automation
- #18: Automation: db tests, selenium, unit tests
- #19: Happy agile team with satisfied customers with collaboration and smart exploratory testing, not automation Can stick to commitment, but then we promise even less to be safe.