Anton Muzhailo - Practical Test Process Improvement using ISTQB
- 1. Practical Test Process Improvement using ISTQB Anton Muzhailo
- 2. Speaker info Anton Muzhailo, Ukraine Senior Python Automation Engineer, GlobalLogic • ISTQB Certified Test Manager • ISTQB Certified Trainings Coach • 3 years of mentoring experience, 200+ students in/muzhailo/
- 3. General Question What is the value of ISTQB knowledge and certifications? - Common vocabulary of testing terms - Test process consistency improving - Test effectiveness improving - Common & best practices - Recognition as expert - Be on the same level as many others
- 4. True Question What is the TRUE value of ISTQB knowledge and certifications? - Clear and visible ROI is a great argument for management - Calculated real amount of time and resources saved - Wider view-angle always leads to better risk-management - Starting “from scratch” always require some classical basis to start - Adapting classical approaches to real cases using your expertise - Lots of approaches and tools cannot become obsolete
- 5. Quality Assessment Problem: Your are a Test Manager of a team on a new project moved from another country. There are 500 test cases written. No formal specification available. • How to see “the big picture”? • What is the ~ test coverage? • How to start? • Where to focus the testing effort?
- 6. Quality Metrics Metrics that are not valid are dangerous! Metrics Qualitative Quantitative - Expensive - Risky - Hard to do well - Suffer from own qualitative problems - Could give a false impression - Can’t be used everywhere - Could be invalid - Hard to choose valid metrics
- 7. Is this metric valid? 𝑇𝑒𝑠𝑡 𝑐𝑎𝑠𝑒 𝑝𝑎𝑠𝑠 𝑟𝑎𝑡𝑖𝑜 = # 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠 𝑝𝑎𝑠𝑠𝑒𝑑 # 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠 𝑟𝑎𝑛 Pretty popular, huh?
- 8. Is this metric valid? 𝑇𝑒𝑠𝑡 𝑐𝑎𝑠𝑒 𝑝𝑎𝑠𝑠 𝑟𝑎𝑡𝑖𝑜 = # 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠 𝑝𝑎𝑠𝑠𝑒𝑑 # 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠 𝑟𝑎𝑛 - How about other 238459293465013475 tests, which were not run or even defined? Pretty popular, huh?
- 9. Is this metric valid? 𝑇𝑒𝑠𝑡 𝑐𝑎𝑠𝑒 𝑝𝑎𝑠𝑠 𝑟𝑎𝑡𝑖𝑜 = # 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠 𝑝𝑎𝑠𝑠𝑒𝑑 # 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠 𝑟𝑎𝑛 - How about other 238459293465013475 tests, which were not run or even defined? Pretty popular, huh? - But we measure test coverage… Seems sufficient!
- 10. Is this metric valid? 𝑇𝑒𝑠𝑡 𝑐𝑎𝑠𝑒 𝑝𝑎𝑠𝑠 𝑟𝑎𝑡𝑖𝑜 = # 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠 𝑝𝑎𝑠𝑠𝑒𝑑 # 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠 𝑟𝑎𝑛 - How about other 238459293465013475 tests, which were not run or even defined? Pretty popular, huh? - But we measure test coverage… Seems sufficient! - Using which metrics?
- 11. Good metrics Once you have two choices with its own pros and cons – you need a hybrid to have both pros’ and none cons © Unicorn from fluffy world
- 12. Good metrics Once you have two choices with its own pros and cons – you need a hybrid to have both pros’ and none cons The popular way of risk mitigation is to split the risk areas to the small parts and manage risks within these parts. © Unicorn from fluffy world Our world require risk assessments
- 13. Such a deep term “Quality” • Must be measured every iteration • Have to be defined in % • Bugs should be represented in Quality’s terms • Such metrics gives more clear vision about product readiness © ISTQB Glossary Quality. The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations.
- 14. Test Coverage Measurement Test cases fully covers the area 1 point Test cases almost covers the area 0.9 pts Test cases covers some of functionality 0.5 pts There are some basic test cases written 0.1 pts There are no test cases written 0 pts 𝑇𝑒𝑠𝑡 𝑐𝑜𝑣𝑒𝑟𝑎𝑔𝑒 = 𝑇𝑜𝑡𝑎𝑙 𝑃𝑜𝑖𝑛𝑡𝑠 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑟𝑒𝑎𝑠 𝑇𝑜𝑡𝑎𝑙 𝑝𝑜𝑖𝑛𝑡𝑠 = 𝑎𝑟𝑒𝑎1_𝑝𝑡𝑠 + … + 𝑎𝑟𝑒𝑎𝑁_𝑝𝑡𝑠 • Area size can vary depending on your expectations and resources • Each area can be divided and measured using the same approach
- 15. Business workflow map Name Q Search 11 Product Page 10 Main Page 9 Cart 9 Login 8 Discounts 8 Supported Countries 5 FAQ 5 More Details 5 Shipping Terms 4 Product Category 4 Checkout 4 Registration 4 Product Subcategory 3 Comment & rate 3 Filters 2 Business map: https://ufile.io/19o89
- 16. Test Coverage Measurement 𝑇𝑒𝑠𝑡 𝑐𝑜𝑣𝑒𝑟𝑎𝑔𝑒 = 9.6 16 = 0.6 ⇒ 60% 𝑇𝑜𝑡𝑎𝑙 𝑝𝑜𝑖𝑛𝑡𝑠 = 9.6 # Name Points 1 Search 0.5 2 Product Page 0.9 3 Main Page 1 4 Cart 1 5 Login 1 6 Discounts 0.1 7 Supported Countries 0.5 8 FAQ 0 9 More Details 0.5 10 Shipping Terms 1 11 Product Category 0.5 12 Checkout 0.9 13 Registration 1 14 Product Subcategory 0.1 15 Comment & rate 0.5 16 Filters 0.1 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑟𝑒𝑎𝑠 = 16 • This approach can be used for progress measurement on every iteration • This approach can show the most and least covered areas • Estimates should be done using review with multiple participants • Correlation between most/least covered areas and areas priority/risk can be established and measured • If each area is still too big to estimate – it can be divided to sub-areas and estimated the same way
- 17. Regression amount estimation Change in Registration shows the regression area immediately Easier to prioritize the tests Faster impact and regression estimation in case of customer issue
- 18. Issue-based metrics # Area Number of Blockers Number of Majors Number of Minors 1 Search 0 8 14 2 Product Page 2 5 11 3 Main Page 0 0 5 4 Cart 2 3 1 5 Login 0 0 7 6 Discounts 0 3 4 7 Supported Countries 0 0 0 … … … … … - So, what’s the quality? - Where we are? - Are we ready to release? - When we will be ready? Accepted definition of ready: • No blockers • All majors have workarounds • No severe Performance or Usability impact - Not yet. - 4 blockers to go - No, 13 majors doesn’t have workarounds - Dunno
- 19. Impact-based metrics # Area Blockers impact Majors impact Minors impact Total impact 1 Search * * * 0.5 2 Product Page * * * 0 3 Main Page * * * 0.9 4 Cart * * * 0.5 5 Login * * * 0.9 6 Discounts * * * 0.9 7 Supported Countries * * * 1 … … … … … No impact (can be released) 1 point Little impact (can be released) 0.9 pts Available for regular testing (can’t be released) 0.5 pts Severe impact (not available for release/testing) 0.1 pts Doesn’t exist or completely blocked 0 pts * - optional 𝑄𝑢𝑎𝑙𝑖𝑡𝑦 = 4.7 7 = 0.67 ⇒ 67% 𝑇𝑜𝑡𝑎𝑙 𝑝𝑜𝑖𝑛𝑡𝑠 = 3.9 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑟𝑒𝑎𝑠 = 7 Quality can also me measured roughly: 𝑄𝑢𝑎𝑙𝑖𝑡𝑦 = # 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛𝑎𝑙𝑖𝑡𝑖𝑒𝑠 𝑟𝑒𝑎𝑑𝑦 𝑓𝑜𝑟 𝑟𝑒𝑙𝑒𝑎𝑠𝑒 𝑡𝑜𝑡𝑎𝑙 𝑄𝑢𝑎𝑙𝑖𝑡𝑦 = 4 7 = 0.57 = 57%
- 20. Test Plan Test Plan. A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice, and any risks requiring contingency planning. It is a record of the test planning process © ISTQB Glossary Problem: Do we really need test plans in Agile? We have JIRA with stories and epics. Why we need an additional test artifact? - Too complex… - Obsolete artifact - Too much additional paperwork - We are still fine without it, don’t care - Nah..my grandma used that in 50s’ - That’s only for mature organizations
- 21. Test Plan artifacts 1. Test Plan Identifier 2. References 3. Introduction 4. Test Items 5. Software Risk Issues 6. Features to be Tested 7. Features not to be Tested 8. Approach 9. Item Pass/Fail Criteria 10. Suspension Criteria and Resumption Requirements 11. Test Deliverables 12. Remaining Test Tasks 13. Environmental Needs 14. Staffing and Training Needs 15. Responsibilities 16. Schedule 17. Planning Risks and Contingencies 18. Approvals 19. Glossary Define what you need. That’s a basis long enough to cover everything’s needed
- 22. Focusing on ‘big picture’ 1 2 3 4 5 60 • 6 new features • New environment • 20 known issues fix Full Regression RELEASE 1 RELEASE 2 Full Regression New features Testing New features Testing Validation sprint Validation sprint Test closure • New major version • Mac support TC design TC update TC update Provides an “expected result” for test monitoring and control activities Shows the planning problems as early as possible Allows complex planning in case of several test teams using Master Test Plan Shows the “big picture”
- 23. Test Control & Monitoring • Such activities can be performed best using the metrics and actions, defined within test plan. • For each metric there should be the action point predicted. • The evaluate method also should be clearly defined • Defect metrics • Pass rate • Employees • Equipment • Licenses • Difference between Planned vs Actual • Time • Money Cost Schedule QualityResources
- 24. Test Control & Monitoring # Metrics Metrics Type Collected Period Evaluate Method Criteria Conclusion 1 Milestone achieving time Schedule Weekly Compare the actual time spent vs planned +/- 20% If deviation is above acceptable values – need an action point 2 Nightly CI set of tests exec. time Resource Daily Check if previous run took less than 10 hrs < 10 hrs 12 hours between each workday. If set of tests reaches 10 hrs – we need another environment set to order 3 Cloud environment cost Cost Weekly Compare the estimated budget draw rate with actual <120% Cost rate depends on traffic and number of users. If deviation is more then 10% we need an action point 4 Duplicate / Not a bug issues Quality Monthly Comparing the actual number with total bugs <10% If more - a time loss. Possibly some engineers need the testing skills improvement or product training 5 Defect clustering Quality Monthly Determine modules, which has many issues in > 5% If more – this modules require attention and effort allocation
- 25. Defects severity estimation Problem: How to choose the good and precise technique for severity estimation no matter how many different testers with different points of view involved? Requirement: Application should support launching in UI mode both from command-line and desktop. Issue: Dropbox doesn’t start using command-line. What is the Severity?
- 26. Defects severity estimation Problem: How to choose the good and precise technique for severity estimation no matter how many different testers with different points of view involved? Requirement: Application should support launching in UI mode both from command-line and desktop. Issue: Dropbox doesn’t start using command-line. What is the Severity?
- 27. Defects severity estimation Severity. The degree of impact that a defect has on the development or operation of a component or system. © ISTQB Glossary Critical Leads to termination. Failed function is unusable and NO acceptable workaround Major Leads to termination. Failed function is unusable and acceptable workaround EXISTS Moderate Doesn’t result in termination. Failed function produces incorrect/incomplete results and NO acceptable workaround Minor Doesn’t result in termination. Failed function produces incorrect/incomplete results and acceptable workaround EXISTS Cosmetic Impacting look and feel of the application or related to enhancement Define your own severity estimation guidelines!
- 28. Root cause analysis Test objective Test condition 1 Test case 1.1 Test case 1.2 Defect 1 Test condition 2 Test case 2.1 Test condition 3 Test case 3.1 Test case 3.2 ??? ??? Defect 2 o What is the root cause of Defect 1? o Maybe several defects have the same root cause? o Why Defect 2 wasn’t covered by any of test cases? o How to improve the test process?
- 29. Questions 5 minutes. You can also ask questions for me in the lounge zone