Th14 Balanced Testing Presentation
0 likes399 views
This document discusses how to achieve the optimal balance of testing through continuous improvement and optimization. It introduces the concept of economic conformance level (ECL) to balance corrective, preventative, and appraisal testing costs. The document outlines a six sigma approach to measure current testing processes, analyze issues, improve through solutions, and control through standardization. Real-life examples are provided to illustrate optimizing testing processes by increasing efficiency, reducing waste, and formalizing practices to move from reactive to balanced testing.
Th14 Balanced Testing Presentation
- 1. Th14 - How Much Testing is Enough? Achieving the Optimum Balance 2 december 2010
- 2. Introduction ISEB Practitioner Lean Six sigma green belt Over 12 years experience within testing area
- 3. Agenda Introduction Test improvement based on ECL* Identify current position Define goal: Next position Define and implement measures * Economic Conformance Level
- 4. Agenda Introduction Test improvement based on ECL Identify current position Define goal: Next position Define and implement measures
- 5. The question is: How to convince the project/organization of the added value of testing?
- 6. Inspired by Joseph Juran Lived from 1904 - 2008 One of the Quality gurus Specialized in managing quality Pioneer of Six sigma The Lean approach Founder of the pareto principle
- 7. Six sigma LSL USL
- 8. Lean approach – avoid wastes Waste Examples of translation to testing Overproduction Good enough testing Waiting Minimize gaps between cause and detect of a problem Transport Deployment of software / Effective Test data and infrastructure management Extra processing Avoid extra testruns and “ Unconscious” redundancy of test Inventory Avoid to much test capacity / Clear and effective reporting Motion Monitoring the total process Defects Prevent instead of detect or correct / Standardize and Simplify human potential Effective use of human capabilities
- 10. Agenda Introduction Test improvement based on ECL Identify current position Define goal: Next position Define and implement measures
- 11. TMM 1. Initial 2. Managed Test policy and strategy Test planning Test monitoring and control Test design and execution Test environment 3. Integration Test organisation Test training program Test life cycle and integration Non-functional testing Peer reviews 4. Management and measurement Test management Software Quality evaluation Advanced peer reviews 5. Optimisation Defect prevention Test process optimisation Quality control Correct Detect Prevent
- 12. Six sigma approach Define -> Improvement goals & cost of poor quality Measure -> current process & establish metrics Analyse -> problems and causes within process & hypothetic testing Improve -> by identifying and piloting solutions & verify solution Control -> by standardization and monitoring
- 13. Reach the balance E. Improve D. Analyse B. Measure F. Control C. Define Detailed A. Define Global
- 14. Agenda Introduction Traditional test improvement Test improvement based on ECL Identify current position Define goal: Next position Define and implement measures
- 15. Software quality Maturity levels Corrective costs Preventive and Appraisal costs Total costs on software quality Reactive Passive Balanced (Over)Active
- 16. Quality measures Corrective Measures taken to correct the failure itself, correct damage and reduce the cost/damage of failures Measurement: Categories the failures on cost: Effort needed to correct the failure + Cost needed to correct the loss of image + Financial damage (downtime, lost of turnover) Appraisal Measures taken to detect the failures, before going live Cost: Effort and other costs needed to find the bugs Benefit: Corrective cost saved on the bugs Preventive Measure taken to prevent failures/bugs Cost: Effort needed to prevent the bugs Benefit: Corrective cost saved on the bugs
- 17. Calculation ECL: Corrective is Appraisal plus Preventive Corrective: # of issues * average corrective costs Direct costs to correct the issues Costs on non productive employees Financial costs (lost of image, extra letters to be sended, lost of turnover, wrong calculations, downtime of website, etc.) Detective (Appraisal) and Preventive: # of issues prevented/detected * costs per issue To be calculated per TEST TYPE and issue CATEGORY !
- 18. Example * Defect Detection Percentage Start N<8 AND MAN=TRUE M>6 OR L<65 End Statement coverage (SC) Condition coverage (CC) Decision coverage (DC) Modified Condition/Decision coverage (MCDC) Branche Condition combination coverage (BCCC) # Average DDP* 2 25 – 55 % 2/3 60 – 80 % 3 60 – 80 % 4 80 – 85 % 16 90 – 95 %
- 19. Agenda Introduction Test improvement based on ECL Identify current position Define goal: Next position Define and implement measures
- 20. Optimizing the balance Reactive Passive Balanced (Over)Active
- 21. Optimizing the balance Move towards optimum via Increasing/decreasing test coverage Introduction/reduction of appraisal measures Introduction/reduction of preventive measures: Configuration management Requirement management Change/release management Document review Other measures
- 22. Optimizing the efficiency Reactive Passive Balanced (Over)Active
- 23. Optimizing the efficiency Being more efficient in quality management by Detecting bugs in an earlier stage Testautomation Reducing the time needed to detect the bugs by: Selecting more efficient test specification techniques Reduce the number of test deliverables (or content of test deliverables) Increase the re-usability of testware Decrease the time needed to maintain testware. Optimising the cooperation between the testtypes and involved parties Avoiding 8 wastes
- 24. Real life Example Current position: Corrective about 7,5 FTE (inclusive correction) Preventive/Appraisal: 2 FTE + 1 FTE = 3 FTE Additional problems: No overall process documentation available Large number of ‘Technical’ failures during system test Large number of minor releases Passive
- 25. Optimisation suggestions Some of the suggestions: Formalising the review process (based on evaluations) Introduction of UAT* regression testing (including process descriptions) Formalising Unit testing “ Informalising” System testing Reducing number of releases * UAT = User Acceptance Test Pragmatic and tailor made
- 26. Expected results Higher testcoverage Reduction of number of changes and issues Reduction of number of production failures Reduction of test execution costs Reduction of time needed for breaking in new employees Expected: Appraisal/Preventive: + 0,5 FTE = 3,5 FTE Corrective: - 2,0 FTE = 5,5 FTE Difference: - 1,5 FTE
- 27. Reconsidering the Boehm curve
- 28. Sharing the experience Company/project culture has to be leading Testing less can also be effective Optimizing is measuring (using metrics) Quality is not only testing Being an added value is being proactive and pragmatic Optimizing is a continuous process Prove the added value of testing
- 29. Questions
Editor's Notes
- #6: Martin luther king . Don’t pretent to be as big as he is. Also don’t want to perform an act/speech about the power of testers .. The swedish ladies did better in there fun session about supertesters, than I could ever do. But I have a dream …. About testers considered as equal as other project members. I was wonder … why ….. The added value of testing is still discussed within projects. The introduction of risk based testing improved the value of testing, but we still can not answer the question about what depth of testing is the most efficience depth …. We often stop testing when we are out of budget or time … but did we achieve the best quality level? So the question is not whether we should test of not (we already convinced the project/organisation of the need for testing) … but what is the optimum balance / most optimum value of testing.
- #7: - In the top list of quality gurus next to Deming and Shewhart (Plan do check act cycle), Philip Crosby (Quality is free) - Pareto principle: by doing the job 20%, 80% of the advance of the entire job can be created. Or 80% of the problems are produced by 20% of the causes - Six sigma: performance improvement methodology. Toolbox. Six sigma levels represents 3,4 defects per million opportunities (Five sigma 233 defects) - Lean approach: produce what is needed and avoid wastes. 8 wastes: (overproduction, waiting, transport, extra processing, inventory, motion, defects, waste of human potential)
- #8: Histogram. Lower specification Limit, Higher specification limit. 68.27 percent of any normal population lies between + or – 1 sigma of the average. Sigma is the standard deviation. 95,45 between 2 sigma …. Etc. Define: scope, critical to quality, cost of poor quality, process map Measure: Identify process variables, current measurement system Analyse: Identiy causes of variation, conduct hypothesis testing Improve: identify possible solutions, verify solutions Control: Implement and sustain
- #16: Reactive QA maturity (far left block) On this level mostly corrective measures are present. Failures are found in the production-environment after going life. Next to the direct costs of correcting the failures the costs of marketing to correct the damage/decreased image are also high. The total cost of quality is much too high and can be reduced by 30-50 percent. The organisation/project only reacts on failures that occur. 2. Passive QA maturity (corrective and detective measures at a late stage) The focus within the level lies on finding the failures on the right side of the V-model during dynamic testing (often during the acceptance test). Testing is present but often only in a unstructured and ineffective way. Almost no preventive measures are present. 3. (Over)active QA maturity (far right block) Lots of preventive and detective measures are taken. Lots of test types are present and less defects are found after going life..... But testing and QA are too expensive. Finding a defect will take more time than correcting them. The damage occurred by the remaining defects is almost null. So: Too much detective and preventive measures in relation to the corrective measures. The total costs of quality can be reduced by reducing the number of preventive and detective measures (or by defining more efficient measures). 4. Balanced QA maturity The level in which the right balance between the quality measures is reached and proven via the collected metrics.
- #25: High lead time to train new testers.