Test result abstraction
Download as PPTX, PDF0 likes340 views
The document discusses presenting test results at the right level of granularity for different stakeholders. It recommends: 1) Understanding what information different stakeholders need from test reports for themselves and their own stakeholders. 2) Providing both quantitative metrics and qualitative analysis to avoid abstraction removing important details. 3) Including information on test coverage and confidence to prevent misinterpretation when results are abstracted.
Test result abstraction
- 1. Test Result Abstraction Presenting Test Results with the Right Granularity
- 2. Introduction • Different stakeholders have different needs when it comes to test reports • You cannot give a developer a 1-slide power point and expect results, and you cannot give a line manager a 40 page test report and expect that they read it • You also don’t know how the report will be abstracted once it leaves your hands • It is important to understand what information your direct stakeholders need from you, but it is also important to understand what their stakeholders in turn expect, so that you can try to secure that important information is easily accessible and will not be abstracted away by mistake after the report has left your hands • Listen to your stakeholders, and understand what kind of information they need, both for themselves and for their stakeholders – don’t assume that you know what they need, and in what format they need it
- 3. Granularity Overview High Management High Management Report Management Report Line Management Project Report Project Leads Test Leads & Team Leads Test Report A Test Report B Test Report C Benchmarks & KPI Engineers Found Defects Scripted Test Results Results Automated Regression Exploratory Test Results Other input Test Results
- 4. Hard Numbers vs. Qualitative Judgements • To be valuable test reports need to include both quantitative and qualitative information • Both the actual numbers and the analysis of those numbers • What you put front and centre depends on the stakeholders and what kind of information they are susceptible to • Some stakeholders will only need the qualitative judgement of the tester - Is the product good enough? • But others want to make that decision themselves based on abstracted empirical information such as pass rate, number of defects, and so on because they have other information which is relevant to the decision, such as business or legal information • The tester does not have to whole picture, and should not assume to know better than their stakeholders what the stakeholders need • It is not a testers job to decide what their stakeholders want or need, but to provide the information that meets those wants and needs
- 5. A number that means one thing to an engineer, can mean something completely different for a line manager, due to all the external input that has been provided between the original report and the management report Engineers Project Leads Line Management Test Leads & Team Leads Test Report Team Report Project Report Management Report Why numbers are sometimes important Team Information Project Information Business Information
- 6. Confidence in Results • It is always important to provide your stakeholders with information on the confidence of the results provided • Zero defects found can mean that you tested nothing, or that you have a really good product, or something in between • Without this information, an abstraction of the results can easily be misinterpreted 2000 TCs executed 43 Exploratory Sessions Executed 98% Pass Rate Good Quality Reported Report 0 Critical Defects 0 Critical Defects 200 Man-hours 100 Man-hours Covered all Covered all common functionality and use cases which features with at least Confidence makes up for 98% of one session, focusing regular usage of the on exceptions, error device handling and negative scenarios
- 7. Deception in Results 2000 TCs executed 98% Pass Rate Could mean 40 Failed Test Cases 40 Failed Test Cases 28% Pass Rate for Performance TCs 40 Critical Defects Performance below Several customer customer requirements failing expectations • Include as much information, with the right granularity, as necessary to avoid this deception
- 8. Prioritize Information • When writing test reports, they should be structured so that the stakeholder can easily retrieve the most critical information they need quickly, and dig deeper for more information if necessary • But information that could be important to your stakeholders’ stakeholders should also be easily accessible to secure that all relevant information reaches its intended customers
- 9. Summary • Always include both qualitative and quantitative information – when a test report is abstracted to stakeholder higher up the corporate chain, you do not know which information is relevant to them • Always include enough information to avoid deception, and to give stakeholders confidence in your test reports • Always prioritize information in your test reports so that your stakeholders can pick out the relevant information they need to present to their stakeholders in turn, and to make educated decisions themselves • Talk to your stakeholders to learn what they want and need – don’t assume anything