Put Risk Based Testing in place right now!
Download as PPTX, PDF3 likes1,089 views
- Risk based testing (RBT) is an approach that uses product risks to guide the testing process and reduce risks. It involves identifying product risks, analyzing their likelihood and impact, and using risk levels to prioritize test design and execution. - Implementing RBT involves 10 steps: selecting RBT, identifying stakeholders, identifying risks, extending risk identification, rating impact, rating likelihood, creating a risk matrix, selecting test approaches and techniques, designing test cases with traceability to risks, and risk-based reporting and defect correction.
![05/12/2016 25
Review the list and check against requirements
• Remove the less relevant risk from the list
• What to do with
• A risk but no requirement
• A requirement but no risk
How to keep the most relevant risks in the
list ?
Product Risk Requirement
ID Product Risk Risk Type Requirement
01 Customer cannot start the
transaction at another bank
Functionality Customer shall be able to
perform a transaction at another
bank
02 Customer not issued with receipt
at the end of the transaction
Functionality Customer shall receive a receipt
at the end of the transaction
03 The system is unavailable to the
customer for longer than two
hours
Reliability System shall be available to
customers 24/7
…
…
Example of a set
of product risks
for an after
Pinkster]](https://image.slidesharecdn.com/20161124riskbasedtestingminskriouducosquer-161205061205/85/Put-Risk-Based-Testing-in-place-right-now-25-320.jpg)
Put Risk Based Testing in place right now!
- 1. 05/12/2016 1 Eric RIOU du COSQUER Minsk, November 24th 2015 Put Risk Based Testing in place, right now!
- 2. 05/12/2016 2 You took the risk to attend my presentation Busines Analyst / Product Owner Project Manager Test Manager Functional or Technical Tester Software Quality and Testing consultant Sales person About you
- 3. 05/12/2016 3 Eric RIOU du COSQUER, erdc@certilogtest.com • Business Analysis www.iqbba.org • Member of the executive committee • Requirements Engineering www.reqb.org • Member of the executive committee • International Software Testing www.istqb.org • General Secretary from 2011 to 2015, France Representative afterwards • French Software Testing Qualification Board www.cftl.fr • Manager since 2013 • Test organizations assessment www.tmmi.org • Lead Assessor since 2015 About me
- 4. 05/12/2016 4 The goal is to explain how to implement a Risk Based Testing approach based on PRISMA® (Product RIsk MAnagement) Introduction Risk Management Basics RBT approach What next? Summary Agenda
- 5. 05/12/2016 5 Testing, Risk, and Risk Based Testing Introduction
- 6. 05/12/2016 6 Main activities (after ISTQB) What is testing ? Planning Control Closure Acceptance System Integration Component 1 Analysis and Design Implementation and Execution Evaluation & Reporting Planification Closure Control
- 7. 05/12/2016 7 Definitions (ISTQB) Risk • A factor that could result in future negative consequences; usually expressed as impact and likelihood Product Risk • A risk directly related to the test object Project Risk • A risk related to management and control of the (test) project, e.g. lack of staffing, strict deadlines, changing requirements… What is a risk ?
- 8. 05/12/2016 8 Definition Risk Based Testing • An approach to testing to reduce the level of product risks and inform stakeholders of their status (…). It involves the identification of product risks and the use of risk levels to guide the process What is « RBT » ? (Risk Based Testing)
- 9. 05/12/2016 9 A general risk management approach applied to product risks Risk Management Basics
- 10. 05/12/2016 10 A process with 4 main activities Risk Management Risk assessment Identification Analysis Risk control Mitigation Monitoring What does the general risk management approach consist in ?
- 11. 05/12/2016 11 The result is a list of risks • Advice: 30 risks max ! 1/4 Risk Identification Risks Type Risk 1 Fonctionnal Risk 2 Security Risk 3 Fonctionnal Risk 4: Reliability …
- 12. 05/12/2016 12 Define Likelihood and Impact for each risk, and then a risk level • Risk Level = Probability * Impact 2/4 Risk Analysis Risks Type Likelihood Impact Level Risk 1 Fonctionnal Risk 2 Security Risk 3 Fonctionnal Risk 4: Reliability … … … … …
- 13. 05/12/2016 13 The risk level calculation may be supported by a table 2/4 Risk Analysis
- 14. 05/12/2016 14 Implement actions to reduce the risks • Four mains options 1. Mitigate the risk through preventive measures to reduce likelihood and/or impact 2. Make contingency plans to reduce impact if the risk becomes an actuality 3. Transfer the risk to some other party to handle 4. Ignore and accept the risk, which means doing nothing but wait and see whether the problem occurs or not. • Mitigation with testing • Associate test cases to the risks 3/4 Risk Mitigation
- 15. 05/12/2016 15 Periodically review the risk status , identify new risks and communicate 4/4 Risk monitoring Risks Type Proba. Impact Action Status Level Risk 1 Fonctionnal Risk 2 Security Risk 3 Fonctionnal Risk 4: Reliability … … … … … New Risk
- 16. 05/12/2016 16 A practical approach, step by step RBT approach based on PRISMA® (Product RISk Management)
- 17. 05/12/2016 17 The decision to implement an RBT approach must be made #1 RBT Selection
- 18. 05/12/2016 18 Possible insights Exhaustive testing is impossible The allocated test design and execution time and budget is always reduced The specifications and requirements may not cover the overall set of expected caracteristics The quality and success of a product depend on the final users and customers view How to (be) convice(d) to implement an RBT approach ?
- 19. 05/12/2016 19 The right people to be involved must be identified #2 Stakeholders identification
- 20. 05/12/2016 20 The Test Manager must select different kind of stakeholders Who should be involved in the RBT process ? On the vendor side On the customer side External stakeholder • End user (client of the customer) • Other organizations (regulatory entities,…) • Customer representatives (called “Business”) • Project sponsors • End users (from the customer company) • Installation and Operations personnel • Testers and Quality Assurance staff • Project managers • Business and System Analysts • Developers and architects • DBA • GUI designers • Technical writers • Testers and Quality Assurance staff
- 21. 05/12/2016 21 PRISMA provides a checklist for stakeholders identification Who should be involved in the RBT process ? - Project manager - Business experts - Designers - Testers - Client / sponsor - End users - Usability experts - Operations - Maintenance team - Security - Safety services - Inspectors - Support / helpdesk - Manufacturing - Marketing - Legal - Professional bodies - Special interest groups - Technology experts - Marketing - Customers - System development - Quality assurance - Regulatory bodies
- 22. 05/12/2016 22 A first list of risks must be created #3 Risk identification
- 23. 05/12/2016 23 Different techniques can be combined How to involve the selected stakeholders in the risk identification ? • Requirements based • Interviews • Workshops and Brainstorming sessions Risks Type Risk 1 Fonctionnal Risk 2 Security Risk 3 Fonctionnal Risk 4: Reliability … Same result as above
- 24. 05/12/2016 24 The initial set of product risks must be improved #4 Risk triage or extended identification
- 25. 05/12/2016 25 Review the list and check against requirements • Remove the less relevant risk from the list • What to do with • A risk but no requirement • A requirement but no risk How to keep the most relevant risks in the list ? Product Risk Requirement ID Product Risk Risk Type Requirement 01 Customer cannot start the transaction at another bank Functionality Customer shall be able to perform a transaction at another bank 02 Customer not issued with receipt at the end of the transaction Functionality Customer shall receive a receipt at the end of the transaction 03 The system is unavailable to the customer for longer than two hours Reliability System shall be available to customers 24/7 … … Example of a set of product risks for an after Pinkster]
- 26. 05/12/2016 26 The impact of each risk needs to be rated #5 Impact Rating
- 27. 05/12/2016 27 PRISMA® suggested factors 1. Critical areas (damage, cost and consequences of failure) 2. Visible areas (external visibility of a failure) 3. Most used areas 4. Business importance 5. Cost of rework Which factors shall we consider to rate the impact ? Impact Factor Criticity Visibility … Weight 2 1 … Risk 1 5 3 … Risk 2 3 5 … Risk 3 3 2 … … … … …
- 28. 05/12/2016 28 The likelihood of each risk needs to be rated #6 Likelihood Rating
- 29. 05/12/2016 29 PRISMA® suggested factors 1. Complexity 2. Size 3. Number of changes 4. New technology and methods 5. Inexperience 6. New development vs. re-use 7. Interfacing 8. … Which factors shall we consider to rate the likelihood ? Impact LIkelihood Factor Criticity Visibility … Complexity Size … Weight 2 1 … 1 2 … Risk 1 5 3 … 3 5 … Risk 2 3 5 … 4 1 … Risk 3 3 2 … 2 4 … … … … … … … …
- 30. 05/12/2016 30 Once impact and likelihood are scored, the risks are included in a Matrix #7 Risk Matrix creation
- 31. 05/12/2016 31 Impact and Likelihood are scored for each risk • Each risk may be rated by different profiles • Impact: business skills • Likelihood: technical skills How to visualize the risk distribution ? Impact Probabilité Factor Criticity Visibility VALUE Complexity Size VALUE Weight 2 1 na 1 2 na Risk 1 5 3 13 3 5 13 Risk 2 3 5 11 4 1 6 … … … … … … …
- 32. 05/12/2016 32 Each risk will be positioned in a matrix What is the Product Risk Matrix ? IIV II IIII IIIII LikelihoodofDefects (TechnicalRisks) Impact of Defects (Business Risks) 3 3 15 15 R1 R2 R3 R4 R5
- 33. 05/12/2016 33 IIV Consider the following advice 1. Avoid the central circle 2. Try not to have all the risks in the same areas 3. Add a fifth area for safety-critical applications How to ensure a right distribution of the risks ? IIV II IIII IIIII Likelihood Impact3 3 15 15 R1 R2 R4 R5 R5 R7 R6
- 34. 05/12/2016 34 The test approach will be based on the risk distribution #8 Test approach and Test techniques selection
- 35. 05/12/2016 35 Impact and Likelihood help you focus on the right level(s) How to allocate the test effort on the different levels ? IIV II IIII IIIII Likelihood Impact3 3 15 15 component and Integration level test (focus on technical risk) system and acceptance level test (focus on business risk)
- 36. 05/12/2016 36 This question should be adressed for each test level How to select the right techniques and define the associated coverage goals ? IIV II IIII IIIII Likelihood Impact3 3 15 15 Example for the component level Decision coverage (90%) Code inspection Instruction coverage (90%) Instruction coverage (70%)
- 37. 05/12/2016 37 This question should be adressed for each test level How to select the right techniques and define the associated coverage goals ? IIV II IIII IIIII Probabilité Impact3 3 15 15 Use Case (incl alternative paths) Decision table Use Case (main path) Equivalence partitioning Use Case (incl alternative paths) Equivalence partitioning Use Case (main path Exploratory testing Example for the acceptance level
- 38. 05/12/2016 38 The traceability from risks to test cases is implemented # 9 Test Design… and Execution
- 39. 05/12/2016 39 Use the traceability How to reach the final Risk Based Test Execution step ? Product Risk Requirement Test Cases Test Execution Results Defects
- 40. 05/12/2016 40 The risk likelihood and impact must be reviewed based on the test execution results #10 Risk Based reporting and Defect correction
- 41. 05/12/2016 41 Update it ! What to do with the Product Risk Matrix Product Risk Requirement Test Cases Test Execution Results Defects Defects Likelihood is increased Passed test cases Likelihood is decreased New risks ?
- 42. 05/12/2016 42 Increase your knowledge in RBT and implement it right now! What next ?
- 43. 05/12/2016 43 And at any time! RBT is everywhere in the test process
- 44. 05/12/2016 44 The Best seller about RBT • ISBN 9789490986070 Sources
- 45. 05/12/2016 45 With your own Excel file or the PRISMA® tool The method can be tooled
- 46. 05/12/2016 46 ISTQB Advanced Level Test Analyst Syllabus Additional Sources • http://www.istqb.org/downloa ds/send/10-advanced-level- syllabus-2012/53-advanced- level-syllabus-2012-test- analyst.html
- 48. 05/12/2016 48 Eric RIOU du COSQUER erdc@certilogtest.com www.certilogtest.com Thank you !
Editor's Notes
- #11: Four different options for Risk Mitigation Once a risk has been identified and analyzed, there are four main possible ways to handle that risk: Mitigate the risk through preventive measures to reduce likelihood and/or impact Make contingency plans to reduce impact if the risk becomes an actuality Transfer the risk to some other party to handle Ignore and accept the risk, which means doing nothing but wait and see whether the problem occurs or not.
- #12: Les testeurs ne sont pas les seuls à pouvoir ou devoir identifier des risques, bien au contraire
- #13: Les testeurs ne sont pas les seuls à pouvoir ou devoir identifier des risques, bien au contraire
- #15: Réduction avec le test: Oui mais Comment??? Voir Prisma !
- #19: Exhaustive testing is impossible Focuss on the most risky areas The allocated test design and execution time and budget is always reduced Be ready to cover and test a subset of the full product’s caracteristics The specifications and requirements may not cover the overall set of expected caracteristics Identifiy additional caracteristics to be tested, from the risk identification The quality and success of a product depend on the final users and customers view Take their point of view into consideration
- #21: Les testeurs ne sont pas les seuls à pouvoir ou devoir identifier des risques, bien au contraire
- #22: Les testeurs ne sont pas les seuls à pouvoir ou devoir identifier des risques, bien au contraire
- #26: Souvent, un risque ou niveau de risque peut être associé à une exigence ou à un ensemble d’exigences Un risque, pas d’exigences Ajouter exigences ou enlever le risque Une exigence, pas de risque Ajouter un risque ou envisager de supprimer l’exigence Exemple: Exigence: Le système doit être disponible 24/24 Risque: Le système est indisponible pendant plus de 1 heure
- #28: Criticité: exemple DO 178B Criticité: 1 à 5 Visibilité: 1 à 5
- #30: Criticité: exemple DO 178B Criticité: 1 à 5 Visibilité: 1 à 5
- #36: Des tests seront mis en face des risques. En fonction du risque ces tests seront positionnés à différents niveaux
- #37: Ne pas oublier la notion de couverture Prisma donne d’autres conseils comme: niveau d’expertise des développeurs…
- #38: Ne pas oublier la notion de couverture