Exploratory Testing Kari Kakkonen BTD 2017
Download as PPTX, PDF1 like393 views
The document discusses exploratory testing, highlighting its essential features, such as simultaneous planning and execution, iterative learning, and limited documentation. It illustrates the approach with insights from industry expert Kari Kakkonen, emphasizing its benefits in agile environments, including enhanced coverage and rapid feedback. Additionally, it outlines techniques, tools, and the evolving role of exploratory testing in future software development cycles.
![Testing dashboard as a test report - an example
33© Copyright Knowit Oy 2015 | Confidential
Test area Workload Coverage Quality
level/risks
Comment
Main page !Interrupted High,
5h
Very high [all
parts + stress
tests etc..]
49: 1435, 36:
1469,
42: 1501
wait for more
pictures of
user interface
Shopping
cart
!Started
High, 2h (reserved
4h)
Low [main parts
to testing] (High)
81: 1425
[probability 9 x
effect 9; error
number. 1425]
Order !Done
6h (reserved 4h)
High [all parts]
Feedback !Not done
Low (reserved 1h)
Low [main parts
to testing]
Reference: James Bach](https://image.slidesharecdn.com/exploratorytestingkarikakkonenbtd25102017-171027145603/85/Exploratory-Testing-Kari-Kakkonen-BTD-2017-33-320.jpg)
Exploratory Testing Kari Kakkonen BTD 2017
- 1. BTD and BNTQB Test Summit in Brussels October 25, 2017 Kari Kakkonen, Knowit Oy, FiSTB, ISTQB @kkakkonen Exploratory testing – basics, experiences and future? Knowit 1
- 2. Kari Kakkonen ROLES • Knowit Oy, Director/Quality and Competences, Lead Consultant, Trainer and Coach • Treasurer of ISTQB Executive Committee • Chairman of Finnish Software Testing Board (FiSTB) • Auditor of Robot Framework Foundation. ACHIEVEMENTS • Influencing testing since 1996 • Ranked in 100 most influential IT persons in Finland (Tietoviikko magazine) • Great number of presentations in Finnish and international conferences • TestausOSY/FAST founding member. • Co-author of Agile Testing Foundations book • Regular blogger in Tivi-magazine EDUCATION • ISTQB Expert Level Test Management Full & Advanced Full & Agile Tester certified • DASA DevOps Fundamentals, Scrum Master and SAFe certified • SPICE provisionary assessor certified • M.Sc.(Eng), Helsinki University of Technology (present Aalto University), Otaniemi, Espoo • Marketing studies, University of Wisconsin-Madison, the USA. BUSINESS DOMAINS • Wide spread of business domain knowledge • Embedded, Industry, Public, • Training, Telecom, Commerce, • Insurance, Banking, Pension SERVICES • ISTQB Advanced, Foundation and Agile Testing + Knowit Quality Professional • DASA DevOps Fundamentals • Quality & Test process and organization development, Metrics • Agile testing, Scrum, Kanban, Lean • Leadership • Test automation, Mobile, Cloud, DevOps • Quality, Cost, Benefits. Twitter: @kkakkonen LinkedIn: fi.linkedin.com/in/karikakkonen/ Knowit 2
- 4. What is it? Knowit 4
- 5. Characterization of exploratory testing • ”Planning and execution of testing is done at the same time” (After James Bach) • Test cases are not necessarily documented even afterwards (Cem Kaner) • In exploratory testing, test design and test execution occur at the same time, guided by a prepared test charter. (ISTQB Agile Tester syllabus) • Testing is done iteratively piece by piece • Continuous learning and interpretation • Utilization of knowledge gained from experience 5© Copyright Knowit Oy 2015 | Confidential
- 6. Exploratory testing - terms • Test Charter • Divide your testing areas and assign missions to each • Testing area: a bunch of functionalities • Exploration • Adventure may go to sidetrack as long as you come back to mainroad again (Kaner) • Testing session • Duration about ½ - 2 hours • Why? • Time span of concentrated work is about 20 minutes • Getting back to work takes about 20 minutes 6© Copyright Knowit Oy 2015 | Confidential
- 7. Learning in exploratory testing 7© Copyright Knowit Oy 2015 | Confidential Testing Opinion-forming Reporting Designing actions Observations After reference: Psychology of Usability, Sinkkonen et al.
- 8. Learning from the system and customers • What has changed or changes frequently? • What do the customers want? • What has been defined in an ambiguous way? • Where do faults cluster? • Weaknesses in the platform or programming language • System dependencies 8© Copyright Knowit Oy 2015 | Confidential Reference: Lessons learned in software testing. Kaner, Bach, Pettichord
- 9. Learn from other testers, designers, from yourself • What kind of errors do certain programmers make and how to report to and communicate with them • What typical errors can there be in the system • What functionalities have been built in a hurry • What have you misunderstood and what is typically misunderstood • How can the system be tested (especially in pair testing!) 9© Copyright Knowit Oy 2015 | Confidential
- 10. Why exploratory testing? Knowledge-based perspective • Exploit the natural diversity of people in testing *) • ”Do not plan for store” • Systematic variation of testing • Quick feedback to designers / developers intensifies learning process *) • Spread the knowledge of a tester of a special area 10© Copyright Knowit Oy 2015 | Confidential *) Reference: Exploratory testing: A multiple case study. Itkonen, Rautiainen
- 11. Why exploratory testing? Testers’ approach • Want to add more test cases and increase the coverage of the tests *) • For defining the degree of automation of tests • Quick overview of the quality *) • Testing the side effects of changes – scripted testing can end up testing only the documented features *) • The problem in regression testing is the selection of test cases, which requires user experience and understanding of the system • When the test documentation can not be written in advance *) 11© Copyright Knowit Oy 2015 | Confidential *) Reference: Exploratory testing: A multiple case study. Itkonen, Rautiainen
- 12. Why exploratory testing? Management approach • Low management costs of test documentation • Finding out the features of a poorly-documented component • Optimizing the productivity of testing? • When the available workload is limited *) • When you want to train the customer support at the same time 12© Copyright Knowit Oy 2015 | Confidential *) Reference: Exploratory testing: A multiple case study. Itkonen, Rautiainen
- 13. Where exploratory testing fits 13 Different styles of testing Adhoc testing Exploratory Testing And Session-Based Test Management High level test cases design and execution Detailed test cases design and execution ET + SBTM High level TC Detailed TC Automation Automated testing and ATDD Exploratory Testing ET Systematicnessoftesting Adhoc ET Scripted testing
- 14. Exploratory testing as part of test strategy 14© Knowit Oy Confidental 2012 Pure (100%) scripted testing Scripted testing Exploratory testing Pure (100%) exploratory testing Where do you put the focus of your own testing?Maybe here?
- 15. How is it done? Knowit 15
- 16. Step by step © Copyright Knowit Oy 2015 | Confidential Plan • Test charter Test sessions • Notes / Log • Bugs Debriefings • Dashboard ”1 hour” Most of the time 5 min regularly
- 17. Exploratory testing in Prague – find a park • Test charter • ”look for green” 1727.10.2017 © Copyright Knowit Oy 2013 | Confidential | Version 1.0
- 18. Exploratory testing in Prague – find a park • Test execution log • Dashboard 1827.10.2017 © Copyright Knowit Oy 2013 | Confidential | Version 1.0
- 19. Exploratory testing in Prague – find a park • Defect report 1927.10.2017 © Copyright Knowit Oy 2013 | Confidential | Version 1.0
- 20. Purpose of test charter(s) • What will be tested? • What documents are available? • What kind of errors are being sought? • Tasks and what test techniques will be used? • What will be expected from test sessions? • A dashboard? • Defect reports? • Some key test cases? 20© Copyright Knowit Oy 2015 | Confidential Reference: A practioner’s guide to software test design. Copeland Ref. Exploratory testing: A multiple case study. Itkonen, Rautiainen
- 21. Test design within test charter • Define the testing areas of the test object • E.g. 20-40 areas covering whole system under test • Divide each area to one or more test sessions • Test charter works as a roadmap for each test session • Define test cases to be documented as a by-product, if any • heuristic: less than 10% of all tests • Write down test ideas and/or high level test cases 21© Copyright Knowit Oy 2015 | Confidential
- 22. Charter classical example Area Coverage and working hours Practice Documents Result possible errors Risks R1. Customer’s all selected items are not added to order, Effect: 20 eur/buyer, probability 5% R2. Order can not be completed after interruption, 5, probability: ?? Main page 100% Path coverage (direct paths) and the most common (80% used) loops 10h Scripting with Functional Tester-tool Main page display description document, navigation map (COMING FROM DEVELOPMENT) All pages and the shopping cart are available Shopping cart 5h Shopping cart- UC.doc (use case) Shopping cart can be used in the same way as a real shopping cart The same product can not be added to shopping cart several times Emptying the shopping cart causes an execption R1 Order ? Order-UC.doc? R2 22© Copyright Knowit Oy 2015 | Confidential Reference: James Bach
- 23. Test Charter example ISTQB 23© Copyright Knowit Oy 2017 | Confidential • Actor: intended user of the system • Purpose: the theme of the charter including what particular objective the actor wants to achieve, i.e., the test conditions • Setup: what needs to be in place in order to start the test execution • Priority: relative importance of this charter, based on the priority of the associated user story or the risk level • Reference: specifications (e.g., user story), risks, or other information sources • Data: whatever data is needed to carry out the charter • Activities: a list of ideas of what the actor may want to do with the system and what would be interesting to test (both positive and negative tests) • Oracle notes: how to evaluate the product to determine correct results • Variations: alternative actions and evaluations to complement the ideas described under activities Reference: ISTQB Agile Tester syllabus
- 24. Charter as an excel 24© Copyright Knowit Oy 2015 | Confidential
- 25. Test charter and test log in mindmap (Xmind) 25© Copyright Knowit Oy 2015 | Confidential
- 26. Documents supporting testing • Charter • List of different testing strategies • Lists of heuristics • List of typical errors • Kaner’s bug taxonomies*) • Legal notices, standards, de facto-standards • Requirements and design documentation of the system • Self made description and models of the system behavior • User guide *) 26© Copyright Knowit Oy 2015 | Confidential *) Reference: Testing Computer Software. Kaner et al. **) Reference: Exploratory testing: A multiple case study. Itkonen, Rautiainen
- 27. Kaner’s bug taxonomies • User interface errors • Error handling • Boundary-related errors • Calculation errors • Initial and later states • Control Flow errors • Errors in handling or interpreting data • Race conditions • Load Conditions • Hardware • Source and version control • Documentation • Testing errors 27© Copyright Knowit Oy 2015 | Confidential Reference: Testing computer software, p. 60 – 64
- 28. Exploratory Testing - Heuristics • A set of heuristics can be applied when testing. • A heuristic can guide the tester in how to perform the testing and to evaluate the results, few examples below 28© Copyright Knowit Oy 2015 | Confidential Heuristics Examples Boundaries: Approaching the Boundary e.g. almost too big, almost too small, at the Boundary CRUD Create, Read, Update, Delete Configuration variations Varying the variables related to configuration e.g. Screen Resolution; Network Speed, Latency, Signal Strength; etc. Interruptions e.g., log off, shut down, or reboot Source: Elisabeth Hendrickson www.testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
- 29. Performance attitude - professional working • Keep the mission, your targets of testing in mind • Write down observations and questions about the system • Report in a disciplined and systematical way • During the execution, write only the most essential test cases and in high level • Test cases can be refined later, if needed 29© Copyright Knowit Oy 2015 | Confidential
- 30. Tests are designed during execution • Design test on the basis of charter its initial test ideas • Define tests from any questions in your mind • A surprising situation may indicate an error: Utilize the surprise effect! • Use heuristics e.g. • ”Backwards thinking”: ”This button saves the definition text. I wonder what other ways are there for saving the text?” 30© Copyright Knowit Oy 2015 | Confidential
- 31. Note taking: test execution logs • Keep a test execution log / notes • Keep track of the tests carried out • Main thing is that executed tests are noted down, not the detail • You may create scripted test cases from some of the tests, add detail • Note what test data you used in your tests • You may record your execution with a suitable tool, only need to add your comments • Add status to your notes e.g. with color-coding • Write down also notes for test session post-mortem and your own learning 31© Copyright Knowit Oy 2015 | Confidential
- 32. Note taking: defect logs • After session work out defect reports for defects you found, based on your notes • In defect reporting, traceability to the requirements must be maintained so that coverage can be evaluated • A well written execution log is the best evidence of the existence of a fault • Report a bug clearly, so that the failure can be repeated • You may use defect reporting systems 32© Copyright Knowit Oy 2015 | Confidential
- 33. Testing dashboard as a test report - an example 33© Copyright Knowit Oy 2015 | Confidential Test area Workload Coverage Quality level/risks Comment Main page !Interrupted High, 5h Very high [all parts + stress tests etc..] 49: 1435, 36: 1469, 42: 1501 wait for more pictures of user interface Shopping cart !Started High, 2h (reserved 4h) Low [main parts to testing] (High) 81: 1425 [probability 9 x effect 9; error number. 1425] Order !Done 6h (reserved 4h) High [all parts] Feedback !Not done Low (reserved 1h) Low [main parts to testing] Reference: James Bach
- 34. Test charter and test log in mindmap (Xmind) 34© Copyright Knowit Oy 2015 | Confidential
- 35. Measuring exploratory testing • The duration of the session • The relative change in the number of test cases by the same tester • Coverage of testing per session • The number of interruptions (Suspension criteria) • Number of rejected defects in defect database • … • Metrics are the eyeglasses of testing that you need in order to be fully aware of the situation and potential problems in testing • It is recommended that you choose metrics that are suitable for the challenges of exploratory testing 35© Copyright Knowit Oy 2015 | Confidential
- 36. Some Tools • Notetaking: • Screenshots With Annotation • Video with Annotation • Intergrated bug reporting • Test Management tool suite add-ons • Micro Focus / HPE Sprinter • QA Symphony qTest eXplorer • Telerik Test Studio Explore • qMetry Voyager • Steps Recorder • Recording Browser extensions • Bug Magnet • Notepad++ (pad tools in general) • Xmind, Mindmup (mindmap tools in general) • Rapid Reporter 36© Copyright Knowit Oy 2015 | Confidential • Test charter planning • Charters • Priorization • Test Management tool suites • Trello • Jira Agile • Micro Focus / HPE Agile Manager • Pivotal Tracker • Excel
- 37. Who can do it? Knowit 37
- 38. Who to recruit? The profile of an exploratory tester • Exploratory testing is particularly well suitable for a person, who… • likes to take risks • is not afraid of changes or new things • is open-minded • sets challenges and goals to themselves • is smart and quick in finding test conditions • ”Pioneer-style” • Everyone can learn to be an exploratory tester 38© Copyright Knowit Oy 2015 | Confidential Reference: Choosing and Managing the Ideal Test Team. Lloyd Roden
- 39. Exploratory testing requires learning skills • Outline the functionality of the system on paper • Aim at understanding • Don’t force yourself to remember facts, use documents • Ask questions about the functionality of the system • Recognize the items on which you need more information 39© Copyright Knowit Oy 2015 | Confidential Reference: Tutkiva oppiminen. Hakkarainen et al.
- 40. Improve your testing thinking skills • Observe your own testing habits • Recognize your own ways of thinking • Learn from misunderstandings and mistakes • Select the testing techniques according to the situation *) • Improve your skills to deduce the states of the system *) 43© Copyright Knowit Oy 2015 | Confidential Adapted from the reference: Tutkiva oppiminen. Hakkarainen et al., *) Reference: Rapid Software Testing. James Bach
- 41. What in future? Knowit 44
- 42. Exploratory future • Exploratory testing will become the norm in agile testing • And agile will be the norm for all development • Testing tools suites will include exploratory testing charters and logging • Coverage will be proven with both last quantities of well-logged exploratory testing and some formally designed test cases • With test automation to make sure the basics work, testers will be free to find important defects with exploratory testing • People will start exploring also with test automation • Keyword driven and data-driven approaches • AI will help with some of exploratory Testing © Copyright Knowit Oy 2015 | Confidential
- 44. 10/27/2017 Knowit 47 Creating digital opportunities