Lecture 06 - 07 - 08 - Test Techniques - Whitebox Testing.pdf
- 2. Why test techniques? • Exhaustive testing (use of all possible inputs and conditions) is impractical – must use a subset of all possible test cases – must have high probability of detecting faults • Need thought processes that help us select test cases more intelligently – test case design techniques are such thought processes 2
- 3. Choosing test technique • The choice of which test techniques to use depends on a number of factors: 3 Tester knowledge and skills Component or system complexity Previous experience with using the test techniques on the component or system to be tested The types of defects expected in the component or system Available tools , documentation Regulatory standards Time and budget Customer or contractual requirements Software development lifecycle model Risk levels Expected use of the software Risk types Type of component or system Test objectives
- 4. What is a testing technique? • a procedure for selecting or designing tests • based on a structural or functional model of the software • successful at finding faults • 'best' practice • a way of deriving good test cases • a way of objectively measuring a test effort Testing should be rigorous, thorough and systematic 4
- 5. Using techniques makes testing much more effective Advantages of Techniques • Different people: similar probability find faults – gain some independence of thought • Effective testing: find more faults – focus attention on specific types of fault – know you're testing the right thing • Efficient testing: find faults with less effort – avoid duplication – systematic techniques are measurable 5
- 6. Categories of Test Design Techniques
- 7. Test Techniques • Black-box Testing Techniques – Specification-based testing • White-box Testing Techniques – Structure-based Testing • Experience-based Testing Techniques – Tester skills, experience and knowledge – Often combined with black-box and white-box 7
- 8. Three types of systematic technique Static (non-execution) • examination of documentation, source code listings, etc. Functional (Black Box) • based on behaviour / functionality of software Structural (White Box) • based on structure of software 8
- 9. Some Test Techniques Static Static Dynamic Dynamic White-box Black-box Functional Non-functional Reviews Reviews Walkthroughs Walkthroughs Desk-checking Desk-checking Flow Data Flow Symbolic Execution Symbolic Execution Definition-Use Definition-Use Statement Statement Branch/Decision Branch/Decision Branch Condition Branch Condition Branch Condition Combination Branch Condition Combination LCSAJ LCSAJ Arcs Arcs Equivalence Partitioning Equivalence Partitioning Boundary Value Analysis Boundary Value Analysis Cause-Effect Graphing Cause-Effect Graphing Random Random Usability Usability Performance Performance Static Analysis Static Analysis Inspection Inspection Flow Control Flow etc. etc. etc. etc. etc. etc. etc. etc. etc. etc. State Transition State Transition 9
- 10. Black box versus white box? Integration Component Acceptance System Black box appropriate at all levels but dominates higher levels of testing White box used predominately at lower levels to compliment black box 10
- 12. White box Software Testing • IEEE Definition “Testing that takes into account the internal mechanism of a system component”. • Internal structure taken into account for testing – Test cases derived from program logic • Concerned with the coverage of program logic • Also referred to as structural testing, clear box testing, and glass box testing (Beizer, 1995) 12
- 13. Black vs. White-box Testing
- 14. Black-box vs. White-box Testing • Black-box Testing • Check conformance with specifications • It scales up (different techniques at different testing levels) • It depends on the specification notation and degree of detail • Do not ‘exactly’ know how much of the system is being tested – Do not detect unspecified task • White-box Testing • It allows you to be confident about test coverage • It is based on control or data flow coverage • It does not scale up (mostly applicable at unit and integration testing levels) • it cannot reveal missing functionalities (part of the specification that is not implemented)
- 15. BBT & WBT in Practice
- 16. Control flow graph (CFG) • Gives an abstract representation of the code • Resembles flow charts • Nodes are blocks of sequential statements • Edges are transfers of control – May be labeled with predicate representing the condition of control transfer
- 18. Basics of CFG: Patterns
- 19. Control Flow Graph – Example
- 20. Control Flow Graph – Exercise (05 mins) 20
- 21. CFG – printCommon( ) Solution 21
- 22. Control Flow Coverage • Depending on the (adequacy) criteria to cover (traverse) CFGs, we can have different types of control flow coverage: – Statement/Node Coverage – Decision Coverage/Branch Coverage – Condition Coverage – Path Coverage
- 23. Adequacy Criterion • Is a way to define a test objective – When to stop testing ? – How long to test ? “A test suit T is said to be adequate for criterion C, when coverage ratio achieves 100% for criteria C” • Given a control flow graph, a possible test criterion C could be to cover all nodes in the graph 23
- 24. Types of control flow coverage • Statement/Node Coverage • Decision Coverage/Branch Coverage • Condition Coverage • Path Coverage
- 25. Statement/Node Coverage • Hypothesis: Faults cannot be discovered if the statements containing them are not executed • Statement coverage criteria: Equivalent to covering all nodes in CFG • For node coverage, simply replace statements with nodes
- 26. Statement Coverage Example • Consider the following code: • Following test case achieves 100% statement coverage – {P = 51, Q = 50 } 26
- 27. Statement Coverage - Exercise1 • Draw the control flow graph • Give test cases to achieve 100% statement coverage .. 27
- 28. Statement Coverage – Issues (1) • Consider the following code block : • Statement coverage is achieved when the function is called with input 6000. • What happens when the amount is less than 5000 ? – Potential Null Pointer Exception 28
- 29. Comments on Statement/Node Coverage • Executing a statement is a weak guarantee of correctness, but easy to achieve • Several inputs may execute the same statements • An important question in practice is: – how can we minimize (the number of) test cases so we can achieve a given statement coverage ratio? • Statement coverage is the weakest criteria – Industry struggles in achieving this 29
- 30. Decision Coverage • Statement/Node Coverage • Decision Coverage/Branch Coverage • Condition Coverage • Path Coverage
- 31. Decision/Branch Coverage • Hypothesis: Faults cannot be discovered if all branches of a decision are not executed • Branch coverage criteria: Equivalent to covering all edges in CFG • Branch Coverage = Number of executed branches Total number of branches X 100
- 32. Decision (or branch) Coverage • Select a test set T such that, by executing P for each test case t in T, each edge of P’s control flow graph is traversed at least once • We need to exercise all conditions that traverse the control flow of the program with true and false values • Also known as branch or edge coverage
- 33. Decision Coverage - Example • Consider the following code: • Following test case achieves 100% decision coverage – {P = 51, Q = 50 } – {P =49, Q = 40} 33 It is preferred to meet the criteria with minimum number of test cases
- 34. Decision Coverage • You must write enough test cases that each decision has a T and a F outcome at least once. • Requires calling the function with balance less than 5000 – Null pointer exception will be caught 34
- 35. Decision Coverage - Exercise • Give test cases to achieve 100% decision coverage .. 35
- 36. Decision Coverage - Example void divideSpecial (int a, int b) { if ( a > 100 && b != 0) b = 10; result = a/b; } Statement Coverage - a = 101, b = 1 Decision Coverage - a = 101, b = 1 - a = 80, b =1
- 37. Decision Coverage - Problem void divideSpecial (int a, int b) { if ( a > 100 && b != 0) b = 10; result = a/b; } Decision Coverage - a = 101, b = 1 - a = 80, b =1 Can you see the fault that is not triggered? What happens when - a = 80, b =0
- 38. Decision Coverage - Problem void divideSpecial (int a, int b) { if ( a > 100 && b != 0) b = 10; result = a/b; } • There is a possibility that the uncovered fault was triggered by ‘chance’ • However, decision coverage can be achieved without triggering the fault • Testing based on luck vs Testing based on criteria
- 39. Exercise Program finding desired element from an array of items Write test cases to cover decision coverage What if the desired item is last in the list? Create control flow graph
- 40. Decision Coverage - Comments • Decision coverage is better than statement coverage • However, can still miss faults as not all conditions are evaluated • How to improve ?? – Condition Coverage ? 40