Steven K Allott - Effective Testing - SoftTest Ireland
Download as PPT, PDF0 likes550 views
The document discusses effective testing techniques, including risk-based testing and black box techniques like equivalence partitioning, boundary value analysis, decision tables, and state transition testing. It provides examples of how to use these techniques to design test cases and reduce the number of tests needed while improving coverage. Pairwise testing is presented as a technique to further reduce the number of test cases by testing variable combinations rather than all combinations.
Steven K Allott - Effective Testing - SoftTest Ireland
- 1. Presented by Stephen Allott CaseMaker International A Diaz & Hilterscheid group company Effective Testing Using industry standard techniques SoftTest Ireland 16 th February 2006
- 2. Agenda Why do we test software? How do we test effectively? Risk based testing Black Box techniques Pairwise testing Photo: Courtesy of NASA
- 4. Read all about it! US site – down for 3 days, forced to return $30 fee to nearly 260,000 online “fantasy baseball” players. US site down for 2 days, $280,000 in lost sales. Shopping Site: 6-hour downtime: $400,000 in direct costs plus additional in discounts to inconvenienced customers. UK - Concerns over bugs and delays in new air traffic control software UK – Payment clearing system failure at month end delayed wages to many customers UK - Televisions, Cameras sold at very low prices
- 5. Ineffective Testing Creates too many of the wrong tests Duplicates effort unnecessarily Costs too much and takes too long Has an adverse impact on the business as a whole
- 6. Effective Testing Is aligned with the business need Uses a risk-based approach and formal techniques to create fewer tests with increased coverage Facilitates communication with the business Provides pre-production confidence Saves time and money
- 7. Risk Based Testing Identify business risks Assess the impact Severity, if it were to fail Probability of failure Focus more testing on high risk areas Continually measure and monitor the risks
- 8. Test Case Design Techniques – 1 Black box and white box techniques have been proven to be effective Techniques are described in BS7925 Analysis/design separated from execution Improved and demonstrable test coverage
- 9. Test Case Design Techniques – 2 Analysis of the test conditions A formal design process gives you Objectivity Repeatability Consistency Documentation Auditability Traceability Confidence
- 10. Test Case Design Techniques – 3 The design activity alone can find faults Helps drive the need for good requirements Use of formal techniques helps with estimates of overall test effort required Supports test automation
- 11. Black Box Techniques BS7925 techniques Equivalence class partitioning Boundary value analysis Decision tables State transition testing Syntax testing Random testing Other techniques Pairwise Domain analysis Use case testing
- 13. Test Data Copy of live data Realistic data volumes Range of values and inputs
- 14. Equivalence Class Partitioning Can you ever see a need to: try the view from every seat of a cinema or theatre? test drive every car in the showroom? withdraw cash from all the ATMs in your country? If you answered “No, of course not!” what would you do instead, and why?
- 15. Equivalence Class Partitioning Equivalence class or partition Ranges of data Based on an assumption about the software’s behaviour Reduces the number of test cases whilst maintaining reasonable test coverage
- 16. Equivalence Class Partitioning An online car insurance quotation system allows a 30% discount for good drivers aged between 25 and 50 but charges full price if you are under 25. For older careful drivers a 20% discount is allowed and if you are retired we will be happy to quote you a special price Under 17 17 to 24 25 to 49 50 to 64 65 and over £1,000 £700 £800
- 17. Boundary Value Analysis Builds on Equivalence Class technique Assumes errors are made on the boundaries Applicable to value and structure of data where there is a continuous range of values length and character type Generates three test cases per boundary one on the boundary one either side of the boundary the smallest distance away
- 18. Boundary Value Analysis – Example An online car insurance quotation system Under 17 – unable to quote 17 to 24 (premium price) 25 to 49 (30% discount) 50 to 64 (20% discount) 65 and over (price on application) Under 17 17 to 24 25 to 49 50 to 64 65 and over 16,17,18 49,50,51 24,25,26 64,65,66
- 19. Boundary Value Analysis – Example Online Banking (Account balance) Lower boundaries EUR 0.01 (DR) EUR 0.00 (CR) EUR 0.01 (CR) Upper boundaries EUR 999,999.99 (CR) EUR 1,000,000.00 (CR) EUR 1,000,000.01 (exceeds max for this account type)
- 20. Boundary Value Analysis – Example Online Banking (Set up standing order) Boundaries for from date Yesterday (invalid) Today (should be allowed) Tomorrow (should be allowed) Boundaries for until date 12 months from today (ok) 12 months less 1 day (ok) 12 months and one day from today (invalid)
- 21. Decision Tables Another intuitive black box technique Express complex business rules in a table Cause-Effect graphing helps with the analysis Define the combination of conditions (causes) Define the resulting actions (effects) Easy to create test cases
- 22. Example – Train Tickets Pay £30 Pay £20 Pay £15 Pay £10 Actions N Y N Y Travels after 9 am N N Y Y Holds a railcard Rule 4 Rule 3 Rule 2 Rule 1 Conditions
- 23. Example – Train Tickets Test Cases Pay £30 Pay £20 Pay £15 Pay £10 Expected results N Y N Y Travels after 9 am N N Y Y Holds a railcard Test 4 Test 3 Test 2 Test 1 Inputs
- 24. Example – ATM Reject Reject Reject Auth Actions N Y N Y Enter valid PIN N N Y Y Enter valid card Rule 4 Rule 3 Rule 2 Rule 1 Conditions
- 25. Example – ATM “don’t care” Reject Reject Auth Actions --- N Y Enter valid PIN N Y Y Enter valid card Rule 3 Rule 2 Rule 1 Conditions
- 26. Example – File Backup Y Y Y N Stick inserted Works Fails Fails Fails Actions Y N --- --- Sufficient space Y Y N --- Stick formatted Rule 4 Rule 3 Rule 2 Rule 1 Conditions
- 27. State Transition Testing Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm.
- 28. State Transition Testing Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm.
- 29. 0-Switch coverage Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm. S2 S4 S1 S1 S3 S2 Finish State D AD T T AT D Expected Output DS R CM TS R CM Input S4 S2 S2 S3 S1 S1 Start State 6 5 4 3 2 1 Test Case
- 30. 1-Switch coverage Source: Standard for software component testing, working draft 3.3 http://www.rmcs.cranfield.ac.uk/~cised/sreid/BCS_SIG/index.htm. S4 S1 S2 S3 S2 S3 S2 S1 S4 S1 Finish State AD T D AT D AT D T AD T Exp. Output R CM DS R CM R CM TS R CM Input S2 S2 S4 S1 S1 S1 S1 S3 S2 S2 Next State D D AD T T T T AT D D Exp. Output DS DS R CM CM TS TS R CM CM Input S4 S4 S2 S2 S2 S3 S3 S1 S1 S1 Start State 10 9 8 7 6 5 4 3 2 1 Test Case
- 31. Pairwise Testing Acknowledgement This session on pairwise testing techniques is based on the book by Lee Copeland: A practitioner’s guide to software test design, published by Artech House, 2004, ISBN 1-58053-791-X
- 32. Pairwise Testing Pairwise is based on testing pairs of combinations of variables rather than all combinations It reduces the number of test cases Many faults are either Single mode Double mode There are two recognised methods: Orthogonal arrays Allpairs algorithm
- 33. Pairwise Example Consider a simple database enquiry screen: It has 4 fields Each field has three possible choices Giving 3 X 3 X 3 X 3 = 81 possible combinations to test With Pairwise Testing You can reduce this to 9 tests!
- 34. Pairwise Example Holiday destination UK, USA, Europe Price range (for family of 5) Budget: Under £2000 Medium: £2000 to £5000 Expensive: Over £5000 Standard of hotel Basic, 4 star, luxury Availability Now: Within 7 days from today Soon: After a week but up to one month from today Later: Later than one month from today
- 35. Orthogonal Array L 9 (3 4 ) This orthogonal array has 9 rows, 4 columns and three ‘levels’ per column 1 2 3 3 Row 9 3 1 2 3 Row 8 2 3 1 3 Row 7 2 1 3 2 Row 6 1 3 2 2 Row 5 3 2 1 2 Row 4 3 3 3 1 Row 3 2 2 2 1 Row 2 1 1 1 1 Row 1 D C B A
- 36. Orthogonal Array L 9 (3 4 ) Each variable becomes a column in the orthogonal array 1 2 3 3 Row 9 3 1 2 3 Row 8 2 3 1 3 Row 7 2 1 3 2 Row 6 1 3 2 2 Row 5 3 2 1 2 Row 4 3 3 3 1 Row 3 2 2 2 1 Row 2 1 1 1 1 Row 1 When Hotel Price Where
- 37. Orthogonal Array L 9 (3 4 ) For each column we fill out the possible choices 1 2 3 Europe Row 9 3 1 2 Europe Row 8 2 3 1 Europe Row 7 2 1 3 US Row 6 1 3 2 US Row 5 3 2 1 US Row 4 3 3 3 UK Row 3 2 2 2 UK Row 2 1 1 1 UK Row 1 When Hotel Price Where
- 38. Orthogonal Array L 9 (3 4 ) Now the array is complete Now 4 star High Europe Row 9 Later Basic Medium Europe Row 8 Soon Luxury Budget Europe Row 7 Soon Basic High US Row 6 Now Luxury Medium US Row 5 Later 4 star Budget US Row 4 Later Luxury High UK Row 3 Soon 4 star Medium UK Row 2 Now Basic Budget UK Row 1 When Hotel Price Where
- 39. Orthogonal Array L 9 (3 4 ) Each row becomes a test case – you can add more if you wish Now 4 star High Europe Test 9 Later Basic Medium Europe Test 8 Soon Luxury Budget Europe Test 7 Soon Basic High US Test 6 Now Luxury Medium US Test 5 Later 4 star Budget US Test 4 Later Luxury High UK Test 3 Soon 4 star Medium UK Test 2 Now Basic Budget UK Test 1 When Hotel Price Where
- 40. How to use Orthogonal Arrays Identify the variables Determine the number of choices for each Locate an orthogonal array of the right size Map the testing problem onto the array Construct the test cases
- 41. What size array do we need? How many possible combinations are there? Choices Variable NT, XP, Linux Server O/S IIS, Apache, WebLogic Server W95, W98, W2000, NT, XP, XP-PRO Operating systems None, Real player, Media Player Plug-ins IE5, IE6, …. 8 in total Browsers
- 42. Using Orthogonal Arrays We require an array with 5 columns Column 1 needs 8 levels (browsers) Column 2 would need 3 levels (plug ins) Column 3 would need 6 levels (O/S) Columns 4 & 5 would need 3 levels (Server, Server O/S) Ideally we need 8 1 6 1 3 3 but it doesn’t exist! The next largest is L 64 (8 2 4 3 ) 2 columns of 1 – 8 3 columns of 1 – 4 So rather than 1,296 tests we need only 64 tests! Remember - one test case per row
- 43. Allpairs Algorithm Devised by James Bach Generates fewer tests In our example, it generates just 48 tests compared to 64 from using orthogonal arrays Consider 100 variables, 2 choices Using orthogonal arrays you need 101 tests Using all pairs you need only 10 tests
- 44. Pairwise Testing – Summary Test all pairs rather than all combinations With 4 variables and 3 choices for each you would require 81 test cases whereas you can cover all pairwise input combinations with only 9 tests For 13 variables and 3 choices for each you would require1,594,323 test cases whereas you can cover all pairwise input combinations with only 15 tests Pairwise guides us to help find Single mode defects Double mode defects You should always add any known risky combinations
- 45. Effective Testing – Summary Effective Testing uses A risk-based approach Formal test case design techniques The benefits are Reduced number of test cases Repeatable well documented tests Improved planning, estimation and communication Facilitates earlier involvement with the business Find faults earlier Give you a consistent approach to testing Save time and money Enables test automation
- 46. After all, what is quality?
- 47. Presented by Stephen Allott CaseMaker International A Diaz & Hilterscheid group company Effective Testing Using industry standard techniques SoftTest Ireland 16 th February 2006