Exploratory testing by Rik Marselis
1 like510 views
The document discusses exploratory testing and provides an overview of the speaker's approach. Some key points include: exploratory testing combines experience, structure, and techniques; it involves chartered sessions with logging and debriefing; the tandem approach uses two testers; and it simultaneously designs, executes, and learns from tests. The goal is to gain confidence through a flexible process that fits agile and DevOps environments.
Exploratory testing by Rik Marselis
- 1. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 1 Exploring software; Combine experience, structure& techniques
- 2. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 2 | Todays presentation Our flavour of Exploratory Testing A little theory Some exercises and dynamic testing Exploratory Testing © Sogeti 2016 2 This little fellow is R0B3. He will assist us in this session.
- 3. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 3 Our flavour of exploratory testing.
- 4. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 4 | E.T. sources: Exploratory Testing © Sogeti 2016 4 Cem Kaner first used “Exploratory Testing” in 1984 Good books: “Explore it!” by Elisabeth Hendrickson “Exploratory Software Testing” by James Whittaker Website: www.TMap.net And many more sources…
- 5. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 5 | E.T. has many flavours, this is our flavour: Exploratory Testing © Sogeti 2016 5 • Focus on confidence (risk-based) • Structured (charter, log, debriefing) • Session-based & Timeboxed (not too short, not too long) • Tandem-approach (two people) • Combine experience-based and coverage-based testing • Simultaneous test design, test execution and learning • Flexible (fit for Agile and DevOps) • Prepared (test ideas, testing tours) • Tools (heuristics, checklists, test design tooling) • Serious Fun !!
- 6. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 6 Focus on confidence Confidence
- 7. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 7 | Testing = Minimizing difference between expected and delivered Exploratory Testing © Sogeti 2016 7 Expected Delivered Business value achieved Business value missed Product risks materialized and money wasted
- 8. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 8 | Definition of testing Exploratory Testing © Sogeti 2016 8 TMap® (Test Management approach): Testing is a process that provides insight into, and advice on, quality and the related risks
- 9. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 9 | Testing contributes to gaining CONFIDENCE © Sogeti 2016 9 Quality Risks Confidence Quality attributes Analysis and mitigation
- 10. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 10 | Definition of exploratory testing Exploratory Testing © Sogeti 2016 10 Elisabeth Hendrickson: Exploratory Testing is simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next.
- 11. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 11 | What is the main deliverable of testing? Exploratory Testing © Sogeti 2016 11 Go No Go ClientTester
- 12. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 12 | How do you get the test report? Investigate the situation Prepare a test (input & expected result) Execute a test (compare the actual result) Use the outcome to report Testing à la façon TMap © Sogeti 2016 12 Use a testing method
- 13. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 13 | Purpose of exploratory testing Exploratory Testing © Sogeti 2016 13 Fast Feedback Learning Revealing the unexpected Gaining confidence
- 14. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 14 Tandem- approach.
- 15. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 15 |15 How testing is often perceived Two IT people are cycling up a hill on a tandem-bike. When they finally reach the summit the one in front, the developer, says: ‘It was hard but we made it!’ Then the one at the back, the tester, comments: ‘Yes and I hope you appreciate that I was using the brakes all the time because otherwise we would have rolled down the hill backwards’ Make sure you are a tester that helps cycling up the hill !!
- 16. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 16 | Tandem approach: 2 people Exploratory Testing © Sogeti 2016 16 Why work in pairs? • Two people know more than one • One subject-matter expert and one testing-expert • One does the testing, the other the logging Alternative: one expert and one trainee Beware: not suited for very dominant or very introvert people
- 17. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 17 | Why 2 people see more than 1 Exploratory Testing © Sogeti 2016 17
- 18. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 18 | Why 2 people see more than 1 Exploratory Testing © Sogeti 2016 18
- 19. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 19 Structured (charter, log, debriefing)
- 20. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 20 | Information / Conclusion / Advice Session log Organize with charters: 3 major activities Exploratory Testing © Sogeti 2016 20 Create charters Execute charter Debriefing Backlog of charters Session log Session log Information / Conclusion / Advice Information / Conclusion / Advice
- 21. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 21 | Our pragmatic simple charter template Exploratory Testing © Sogeti 2016 21 Charter Log (notes) De- briefing Test ideas: e.g. testing tours, test data, etc.
- 22. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 22 | Determine test units, assign test charters Exploratory Testing © Sogeti 2016 22 Someone organizes exploratory testing upfront by creating charters (e.g. a test manager, scrum master, team member, etc.) Divide the test object in logical test units Create one or more charters per test unit This may involve any kind of risk analysis (e.g. risk poker) Create a few extra “empty” charters that can be assigned based on conclusion and advice in the debriefing.
- 23. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 23 Prepared (testideas, testing tours)
- 24. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 24 | Test ideas Inspired by: Elisabeth Hendrickson Exploratory Testing © Sogeti 2016 24 A test idea is any useful thought, piece of data, technique, heuristic or whatever that you write down on your charter so that during your test session you have an abundance of possibilities to vary your testing. Make sure you also think of non-functional quality characteristics!
- 25. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 25 | Testing tours Exploratory Testing © Sogeti 2016 25 Landmarks of Amsterdam Rijksmuseum Dam square Red lights Van Gogh museum
- 26. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 26 | Testing tours Exploratory Testing © Sogeti 2016 26 James Whittaker distinguishes many different possible testing tours. Examples: The landmark tour: think of all the “landmarks” that you must have seen during your exploratory session, while travelling between these landmarks you explore the rest of your test object. The supermodel tour: Only the outside matters, focus testing on the look-and-feel. The intellectual tour: Prepare “hard questions” for the test object, e.g. if you must open a file, find the hardest file to open. (for more tours read: “Exploratory Software Testing” by James Whittaker)
- 27. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 27 | Exercise: test ideas for line-following robot Exploratory Testing © Sogeti 2016 27 R0B3 is a robot with a light sensor. It can follow a line It will stop if it detects a red color What test ideas come to your mind?
- 28. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 28 Session-based & Timeboxed.
- 29. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 29 | Running the exploratory testing session Exploratory Testing © Sogeti 2016 29 No distractions No interruptions Timebox: ½ hour – 3 hours (as specified in charter)Charter
- 30. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 30 | Structure: session based & chartered Exploratory Testing © Sogeti 2016 30 Keep the scope of your charter in mind, but keep some room to follow your insights to explore somewhat beyond the scope to learn more. Room for wandering !!
- 31. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 31 | Test log Exploratory Testing © Sogeti 2016 31 Inputs Actions Expected output Actual output Observations, Defects, etc.
- 32. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 32 | Possible debriefing outcomes Exploratory Testing © Sogeti 2016 32 Debriefing Not able to execute test New risks and/or test ideas Return charter to backlog New charter on backlog Information / Conclusion / Advice Information / Conclusion / Advice
- 33. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 33 Combine Experience & Coverage.
- 34. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 34 | Always combine both approaches combine Exploratory Testing © Sogeti 2016
- 35. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 35 | Four groups of Coverage based testing Better know & use one technique of each group than all techniques of only one group Exploratory Testing © Sogeti 2016
- 36. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 36 | Experience based approaches with TMap HD Largely intuitive & ad hoc Structured, chartered Experience brought together Exploratory Testing © Sogeti 2016
- 37. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 37 | Test the triangle program What would you want to test? Why? What is your expected result? What is the actual result? What is your conclusion? © Sogeti 2016 37
- 38. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 38 | Expected outcome is ESSENTIAL Exploratory Testing © Sogeti 2016 38 Be sure FIRST to think of your expected outcome before you execute a test. 1 + 1 = ??? Expected: 2 Actual:
- 39. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 39 Simultanuous test design, test execution, and learning.
- 40. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 40 | Why combine structure and flexibility? Exploratory Testing © Sogeti 2016 40 Exercise The trainer has an object in mind (can be anything in the universe) The participants ask a yes/no question The trainer answers (yes or no only) If a participant knows the object shout it out! Lee Copeland
- 41. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 41 | Exciting… © Sogeti 2016 41
- 42. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 42 |© Sogeti 2016 42
- 43. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 43 Flexible (fit for Agile and DevOps)
- 44. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 44 | Exploration is a feedback loop Exploratory Testing © Sogeti 2016 44 TEST LEARN
- 45. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 45 | Early testing Exploratory Testing © Sogeti 2016 45 Also known as “Reviewing” Also known as “Testing” Start testing as early as possible. Combine static and dynamic testing whenever possible.
- 46. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 46 Tools (heuristics, checklists, test design tooling, test execution tooling, etc.)
- 47. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 47 | Use heuristics to get test ideas Exploratory Testing © Sogeti 2016 47 A heuristic is a rule of thumb, simplification, or educated guess Numbers: - Zero - One - Many - Too many - Too few Selecting: - None - One - Some - Many - All Timing: - Always - Sometimes - Never CRUD: - Create - Read - Update - Delete
- 48. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 48 | Test design techniques are tools ! Exploratory Testing © Sogeti 2016 48 Use coverage based techniques as “test ideas” (thus your start-set of test cases can be made long before …)
- 49. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 49 | Tool: nightmare headline game Exploratory Testing © Sogeti 2016 49
- 50. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 50 Serious fun !!
- 51. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 51 | Bug hunt Exploratory Testing © Sogeti 2016 51 Get a charter Analyse Pair testing one hour Judge evalute defects Score sheet update The “Friday afternoon-sport”: competition in short iterations to determine which team finds the most (real!) bugs (of course there’s a prize!!)
- 52. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 52 | Conclusion source: TMap HD, Niki van Dreumel, ASR Exploratory Testing © Sogeti 2016 52 In the hands of experts, Exploratory Testing is a structured way to measure the quality of the information system in a relatively short time and to reach an objective view on the risks involved.
- 53. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 53 Happy exploration !
- 54. Exploratory Testing, Techtalk with Capgemini Wroclaw (Poland), 7 December 2016 © Sogeti 2016 54 Questions ? Answers ! Discussion