How to reduce your test cases... magically!
Download as PPTX, PDF1 like957 views
The document discusses pairwise testing as a technique for reducing the number of test cases needed to cover all combinations of parameters. It shows that using pairwise testing on a problem with 10 parameters, each with multiple values, reduces the number of test cases from over 36 million to just 97 test cases, cutting the estimated testing time from over 2016 man years to just 1 man day. Pairwise testing provides nearly 100% coverage of defects using far fewer tests than exhaustive testing of all combinations.
How to reduce your test cases... magically!
- 1. Michaël Pilaeten Reducing your testing efforts… magically!
- 2. Magic?
- 3. 3© 2013 CTG, Inc.
- 4. 4© 2013 CTG, Inc. Huh?
- 8. How many test cases do you need?
- 10. 10© 2013 CTG, Inc. Again….
- 15. All combinations How many test cases do you need? No negative tests One test per combination
- 17. 17© 2013 CTG, Inc. 5 x 3 x 3 x 5 x 6 x 7 x 6 x 8 x 8 x 10 = 36.288.000 test cases Writing a test case: 4 minutes Executing a test case: 1 minute 36.288.000 test cases x 5 minutes = 181.440.000 minutes of testing = 378.000 man days of testing = 2016 man years of testing
- 18. 18© 2013 CTG, Inc. One more thing….
- 20. Alternatives?
- 21. 21© 2013 CTG, Inc.
- 22. Automation?
- 23. CHERRY PICKING
- 26. 26© 2013 CTG, Inc. Bug research
- 27. Single mode Double mode
- 28. 28© 2013 CTG, Inc. The solution …
- 29. 29© 2013 CTG, Inc. Pairwise Testing
- 31. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 Condition 1 Condition 2 Condition 3 Condition 4 Condition 5 Binary orthogonal arrays are easy ! 1 1 1 1 1 1 1 1 1 1 Step 1: # TC = (# conditions + 1) Step 2: TC 1 all true Step 3: Left diagonal true Step 4: Remaining untrue 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 Only yes/no options
- 32. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 Condition 1 Condition 2 Condition 3 Condition 4 Condition 5 Binary orthogonal arrays are easy ! 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 A) Each single condition value is tested B) Each couple of condition values is tested single mode defects double mode defects Only yes/no options
- 33. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 Condition 1 Condition 2 Condition 3 Condition 4 Condition 5 Binary orthogonal arrays are easy ! 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 100% certainty of finding all combinatorial defects 25 or 2 x 2 x 2 x 2 x 2 = 32 -> 32 Test Cases Only yes/no options
- 34. Conditions TC 1 TC 2 TC 3 TC 4 TC 5 TC 6 Condition 1 Condition 2 Condition 3 Condition 4 Condition 5 Binary orthogonal arrays are easy ! 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 100% certainty of finding all combinatorial defects 99,8% certainty of finding all combinatorial defects Only yes/no options 32 Test Cases 6 Test Cases
- 36. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities
- 37. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101
- 38. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82
- 39. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82 71
- 40. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82 71 62
- 41. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82 71 62 52
- 42. Age: 3 possibilities Civil status: 5 possibilities Children: 6 possibilities Properties: 6 possibilities Work: 5 possibilities Savings: 8 possibilities Residence: 3 possibilities Ecological: 10 possibilities Wage: 7 possibilities Charity: 8 possibilities We need this* orthogonal array: * or larger L 101 82 71 62 52 32
- 43. Catalogs of Non-Binary orthogonal arrays: http://neilsloane.com/oadir/index.html http://www.york.ac.uk/depts/maths/tables/orthogonal.htm The Taguchi Orthogonal Array Selector: http://www.freequality.org/documents/tools/Tagarray_files/tamatrix.htm
- 46. 46© 2013 CTG, Inc.
- 47. 47© 2013 CTG, Inc. 5 x 3 x 3 x 5 x 6 x 7 x 6 x 8 x 8 x 10 = 36.288.000 test cases Writing a test case: 4 minutes Executing a test case: 1 minute 36.288.000 test cases x 5 minutes = 181.440.000 minutes of testing = 2016 man years of testing Allpairs algorithm: = 97 test cases 97 test cases x 5 minutes = 485 minutes of testing = 1 man day of testing 1 man day versus 2016 man years ?
- 48. 48© 2013 CTG, Inc. 99,8% is not 100% Different types of tests Defect taxonomy
- 49. 49© 2013 CTG, Inc. REDUCE FOCUS
Editor's Notes
- #21: Outsourcing / offshoring Test automation Or… Not testing everything (cherry picking) RRBT
- #25: Risk and requirement based testing: writing and executing the test cases which poses the highest risk of malfunctioning. Pro: - functionalities with highest risk will most likely work Con: - how to make the risk assessment (based on)? - what with low risk areas which are used a lot? (customer satisfaction)
- #28: Research shows that over 99,8% of all defects and incidents reported are either single-mode defects or double-mode defects. Single mode: one mistake in the code results in a defect that shows up during executiuon Double mode: two combined mistakes in the code result in a defect that shows up during execution The change for a Three (or higher) mode defect in the code, after all single mode and double mode defects are found, is below 0,02%.
- #48: 378.000 x less effort