![47
[a] This path would start at the beginning (line 7) --
[b] Decide TRUE at the if() statement (line 13) Executing in foreground
[d] Proceed out of the if() statement (line 16) --
[e] Decide TRUE for the while() predicate (line 17) C
[g] Go into handler that deals with (A..F) --
[k] Proceed out of handler (line 43) --
[m] Proceed back to while loop (line 17) --
[e] Decide TRUE for the while() predicate (line 17) d
[h] Go into handler that deals with (a..f) --
[k] Proceed out of handler (line 43) --
[m] Proceed back to while() loop (line 17) --
[e] Decide TRUE for the while() predicate (line 17) R
[j] Go into handler that deals with non-hex character --
[k] Proceed out of handler (line 43) --
[m] Proceed back to while() loop (line 17) --
[n] Decide FALSE for the while() predicate EOF
[o] Proceed to end of while() (line 44) --
[q] Decide FALSE at the if () statement (line 45) There are hex digits
[r] Proceed to end of function --
abdegkmehkmejkmnoqr Expected output: “Got 2 hex digits: cd”](https://image.slidesharecdn.com/stqalectureslides09-250628153439-49db4723/85/STQA-Lecture-Slides-09-pptxjjjjjjjjjjjjj-47-320.jpg)
STQA Lecture Slides 09.pptxjjjjjjjjjjjjj
- 1. Software Testing & Quality Assurance Prof. Dr. Tariq Javid Graduate School of Engineering Sciences & Information Technology Faculty of Engineering Sciences & Technology Hamdard University Spring 2024
- 2. 2 Static Code Analysis Tools and Techniques
- 3. 3 Topics • Control Flow Testing • Testing Techniques: Statement, Decision, Condition, Path • Cyclomatic Complexity • Data Flow Testing
- 4. 4 Static Analysis • Static analysis encompasses the analytical testing that can occur without executing the software • Software is examined either by a tool or by a person to determine if it will process correctly when it is executed • This static view of the software allows detailed analysis without having to create the data and preconditions that would cause the scenario to be exercised
- 5. 5 Control Flow Testing • An approach to structure-based testing in which test cases are designed to execute specific sequences of events • Various techniques exist for control flow testing, decision testing, condition testing, and path testing, that each have their specific approach and level of control flow coverage
- 6. 6 Control Flow Graphs • Control flow testing is done through control flow graphs, which provide a way to abstract a code module to better understand what it does • Control flow graphs give visual representation of structure of code
- 7. 7 Control Flow Testing Algorithm • The algorithm for all control flow testing consists of converting a section of the code into a control graph and then analyzing the possible paths through the graph • There are a variety of techniques that we can apply to decide just how thoroughly we want to test the code • Then we can create test cases to test to that chosen level
- 8. 8 Statement Coverage • At the low end of control flow testing, we have statement coverage • Common industry synonyms for statement are instruction and code • They all mean the same thing: have we exercised, at one time or another, every single line of executable code in the system? 100% or less!
- 9. 9 Statement Coverage Black-Box Testing ~ 30% • It is possible to test less than that; people not doing white-box testing do it all the time, they just don’t measure it • Remember, thorough black-box testing without doing any white- box testing may total less than 30 percent statement coverage, a number often discussed at conferences urging more white-box testing
- 10. 10 Calculate Statement Coverage • To calculate the total statement coverage actually achieved, divide the total number of executable statements into the number of statements that have actually been exercised by your testing • For example, if there are 1,000 total executable statements and you have executed 500 of them in your testing, you have achieved 50 percent statement coverage • This same calculation may be done for each different level of coverage
- 11. 11 Decision (or Branch) Coverage • This is determined by the total number of all of the outcomes of all of the decisions in the code that we have exercised • Decision testing and branch testing are considered synonymous!
- 12. 12 Condition Coverage • In condition coverage, we ensure that we evaluate each atomic condition that makes up a decision at least once, and multiple condition coverage, where we test all possible combinations of outcomes for individual atomic conditions inside all decisions
- 13. 13
- 14. 14
- 15. 15 Process Block • Figure 2–2, the process block • Graphically, it consists of a node (bubble or circle) with one path leading to it and one path leading from it • Essentially, this represents a chunk of code that executes sequentially: that is, no decisions are made inside of it • The flow of execution reaches the process block, executes through that block of code in exactly the same way each time, and then exits, going elsewhere • The process block has no decisions made inside it
- 16. 16 Junction Point • Figure 2–3, the junction point • This structure may have any number of different paths leading into the process block, with only one path leading out • No matter how many different paths we have throughout a module, eventually they must converge—or exit the module • Again, no decisions are made in this block • All roads lead to it with only one road out
- 17. 17 Decision Point • Figure 2–4, two decision points: one with two outputs, one with five outputs • Decision point is very important; indeed, it’s key to the concept of control flow • Decision point is represented as a node with one input and two or more possible outputs • Its name describes the action inside the node: a decision as to which way to exit is made and control flow continues out that path while ignoring all of the other possible choices
- 18. 18
- 19. 19
- 20. 20
- 21. 21 Where statement coverage successful • Figure 2–7 shows a really simple piece of code and a matching control flow graph • Test case 1 is represented by the gray arrows; the result of running this single test with the given values is full 100 percent statement coverage • Notice on line 2, we have an if statement that compares a to b • Since the inputted values (a = 3, b = 2), when plugged into this decision, will evaluate to TRUE, the line z = 12 will be executed, and the code will fall out of the if statement to line 4, where the final line of code will execute and set the variable Rep to 6 by dividing 72 by 12
- 22. 22 Where statement coverage fails • If we pass in the values shown as test 2 (a = 2, b = 3), we have a different decision made at the conditional on line 2 • In this case, a is not greater than b (i.e., the decision resolves to FALSE) and line 3 is not executed • We still fall out of the if to line 4 and the calculation 72/z is performed • Since z is set to 0, we have 72 divided by 0
- 23. 23 Decision Coverage • Decision or branch coverage looks at decisions level • Every decision has the possibility of being resolved as either TRUE or FALSE • No other possibilities: Binary results, TRUE or FALSE • For those who point out that the switch statement can make more than two decisions, well, conceptually that appears to be true
- 24. 24 Switch Statement • Switch statement is a complex set of decisions, often built as a table by the compiler • The generated code, however, is really just a number of binary compares that continue sequentially until either a match is found or the default condition is reached • Each atomic decision in the switch statement is still a comparison between two values that evaluates either TRUE or FALSE
- 25. 25 Test Cases for Decision Coverage • To get to the 100 percent decision level of coverage, every decision made by the code must be resolved both ways at one time or another, TRUE and FALSE • That means—at minimum—two test cases must be run, one with data that causes the evaluation to resolve TRUE and a separate test case where the data causes the decision to resolve FALSE • If you omit one test or the other, then you do not achieve 100 percent decision coverage
- 26. 26
- 27. 27 Decision Constructs • Table 2–1, you can see a short list of the decision constructs that the programming language C uses • Other languages use similar constructs • Commonality between all of these (and all of the other decision constructs in all imperative programming languages) is that each makes a decision that can go only two ways: TRUE or FALSE
- 28. 28
- 29. 29 for Loop • Looking back at our original example, with just two test cases we did not achieve decision coverage, even though we did attain statement coverage • We did not execute the for loop during the test • Remember, a for loop evaluates an expression and decides whether to loop or not to loop based on the result
- 30. 30 100% Decision Coverage • In order to get decision coverage, we need to test with the value 0 inputted, as shown in Figure 2–8 • When 0 is entered, the first decision evaluates to FALSE, so we take the else path • At line 12, the predicate (is 1 less than or equal to 0) evaluates to FALSE, so the loop is not taken • Recall that earlier, we had tested with a value greater than 0, which did cause the loop to execute • Now we have achieved decision coverage; it took three test cases
- 31. 31 Loop Coverage • Loop testing is an important control flow, structural test technique • If we want to completely test a loop, we would need to test it zero times (i.e., did not loop), one time, two times, three times, all the way up to n times where it hits the maximum it will ever loop • Try to test the loop zero and one time and then test the maximum number of times it is expected to cycle; if you know how many times that is likely to be!
- 32. 32
- 33. 33
- 34. 34 Condition Coverage • A white-box test design technique in which test cases are designed to execute condition outcomes • Atomic condition: It cannot be decomposed • Examples: x, C == D, d != TRUE • Atomic condition does not contain two or more single conditions joined by a logical operator: AND, OR, XOR • Example: D && F
- 35. 35 Untested Atomic Condition • Defects may lurk in untested atomic conditions, even though the full decision has been tested both ways • As always, test data must be selected to ensure that each atomic condition actually be forced TRUE and FALSE at one time or another • Clearly, the more complex a decision expression is, the more test cases we will need to execute to achieve this level of coverage
- 36. 36
- 37. 37 Example • Consider the following pseudo code: if (A && B) then {Do something} else {Do something else} • In order to achieve condition coverage, we need to ensure that each atomic condition goes both TRUE and FALSE in at least one test case each
- 38. 38 Test Cases for the Example if (A && B) then {Do something} else {Do something else} • Test Case 1: A == FALSE, B == TRUE resolves to FALSE • Test Case 2: A == TRUE, B == FALSE resolves to FALSE • Test Case 3: A == TRUE, B == TRUE resolves to TRUE
- 39. 39 Quick Recap
- 40. 40 Path Testing • Path testing is a white-box test design technique in which test cases are designed to execute paths • A path through the software is a sequence of instructions or statements that starts at an entry, junction, or decision and ends at another, or possibly the same, junction, decision, or exit • A path may go through several junctions, processes, or decisions one or more times
- 41. 41 Path Testing Techniques • Path Testing via Control Flow Graphs … Boris Beizer • Path Testing via Cyclomatic Complexity … Thomas McCabe
- 42. 42 Path Testing via Control Flow Graphs • To use control flow graphs to visually pick interesting (and perhaps some uninteresting) paths through the code • General rules for selecting paths: 1. Many simple paths are better that fewer complicated paths 2. There is no conceptual problem executing same code more than once
- 43. 43
- 44. 44
- 45. 45
- 47. 47 [a] This path would start at the beginning (line 7) -- [b] Decide TRUE at the if() statement (line 13) Executing in foreground [d] Proceed out of the if() statement (line 16) -- [e] Decide TRUE for the while() predicate (line 17) C [g] Go into handler that deals with (A..F) -- [k] Proceed out of handler (line 43) -- [m] Proceed back to while loop (line 17) -- [e] Decide TRUE for the while() predicate (line 17) d [h] Go into handler that deals with (a..f) -- [k] Proceed out of handler (line 43) -- [m] Proceed back to while() loop (line 17) -- [e] Decide TRUE for the while() predicate (line 17) R [j] Go into handler that deals with non-hex character -- [k] Proceed out of handler (line 43) -- [m] Proceed back to while() loop (line 17) -- [n] Decide FALSE for the while() predicate EOF [o] Proceed to end of while() (line 44) -- [q] Decide FALSE at the if () statement (line 45) There are hex digits [r] Proceed to end of function -- abdegkmehkmejkmnoqr Expected output: “Got 2 hex digits: cd”
- 48. 48 Determine Coverage • We have control flow diagram, we need to determine the coverage • We create a table to store the paths and facilitate the analysis as to which tests we find interesting and want to run • Table 2–17 include the paths we decide to test, the decisions that are made in those tests, and the actual path segments that are covered in each test
- 49. 49
- 50. 50 Decision Coverage • The decision columns help us determine whether we have decision coverage given the paths we have chosen • After filling in the paths we decide are interesting, we know that we have decision coverage if each decision column has at least one TRUE (T) and one FALSE (F) entry in it
- 51. 51 Decision Coverage for Switch Statement • We have a switch statement in line 18 that has four possible paths through it (f, g, h, and j) • Each of those columns must have at least one entry showing that the path segment has been traversed or we have not achieved decision coverage through that construct
- 52. 52 Statement Coverage • We can ensure that we have statement coverage by ensuring that each code segment has been checked (with an x in this case) in at least one path instance
- 53. 53
- 54. 54 Test Cases • Test Case 1: Test a single digit (0..9) and print it out • Test Case 2: Test a single capitalized hex digit (A..F) and print it out • Test Case 3: Test a single non-capitalized digit (a..f) and print it out • Test Case 4: Test a non-hex digit and show there are no hex digits to convert • Test Case 5: Test entering EOF as first input while running application in background
- 55. 55 Running Application in Background • Note that tests running the application in the background would be interesting, feeding input via a device or file • We know this because we see specific provisions for handling signals in the module • Tests selecting non-trivial strings to test would also be interesting • The five tests just shown should be seen as only the start
- 56. 56 Cyclomatic Complexity • Basis Path Testing • Any software module has a small number of unique, independent paths through it (if we exclude iterations) called basis paths • Structure of code can be tested by executing through small number of paths • All different paths actually consists of simply (re)using the basis paths
- 57. 57 Basis Path Set • Basis path set is the smallest number of basis paths that cover the structure of the code • Theory states that creating a set of tests that cover these basis paths will guarantee us both statement and decision coverage of testing • Basis path has also been called the minimal path for coverage
- 58. 58 Cyclomatic Complexity is a Number • The term cyclomatic is used to refer to an imaginary loop from the end of a module of code back to the beginning of it • How many times would you need to cycle through that outside loop until the structure of the code has been completely covered? • This cyclomatic complexity number is the number of loops needed and—not coincidentally—the number of test cases we need to cover the set of basis paths
- 59. 59 Measuring Cyclomatic Complexity • Measuring cyclomatic complexity by creating a directed control flow graph • This works well for small modules • Tools are generally be used to measure module complexity
- 60. 60 Directed Control Flow Graph • We have nodes (bubbles) to represent entries to the module, exits from the module, and decisions made in the module • Edges (arrows) represent non-branching code statements which do not add to the complexity • If the code does not branch, it has a complexity of one
- 61. 61 Higher Complexity implies More Test Cases • In general, the higher the complexity, the more test cases we need to cover the structure of the code • If we cover the basis path set, we are guaranteed 100 percent of both statement and decision coverage
- 62. 62
- 63. 63 Calculating Cyclomatic Complexity • On the left side of Figure 2–13 we have a function to calculate the greatest common divisor of two numbers using Euclid’s algorithm. • Method 1: Four enclosed regions (R1, R2, R3, R4), represented by R are found in the diagram by noting that each decision (bubble) has two branches that enclose a region of the graph • Method 2: Counting the edges (arrows) and the nodes (bubbles) and applying those values to the calculation E - N + 2, where E is the number of edges and N is the number of nodes
- 64. 64 Method 1: No. of regions (# R) = 4 Cyclomatic Complexity = # R + 1 = 5 Method 2: No. of edges (# E) = 9 No. of nodes (# N) = 6 Cyclomatic Complexity = # E - # N + 2 C = 9 – 6 + 2 = 5
- 65. 65 Simple Method for Large Code • Count the branching and looping constructs and add 1 • If statements, for, while, and do/while constructs, each count as one • Switch/case: Each case block counts as one, default block does not count • In if and ladder if constructs, final else does not count • In example, three if statements and one while statement leads to the cyclomatic complexity: 3 + 1 + 1 = 5
- 66. 66
- 67. 67
- 68. 68 Number of Edges (Arrows): 14 Number of Nodes (Bubbles): 9 Cyclomatic Complexity: 14 – 9 + 2 7
- 69. 69 API • An Application Programming Interface (API) is code which enables communication between different processes, programs and/or systems • APIs are often utilized in a client/server relationship where one process supplies some kind of functionality to other processes • API implements a protocol that specifies a way for a local process to request that a task be done from a remote location and defines the way the results of that task are returned to the local process
- 70. 70 API Testing • As technical test analysts, it is not required that we understand all of the technical details in the implementation of these APIs • We do, however, have to understand how to test them in representative environments and in realistic ways
- 71. 71 Typical API Uses • An organization is likely to use private, partner, and public APIs • Very often a business organization will use APIs to tie together the disparate systems after merging with another company • This can reduce the amount of work needed—rather than trying to merge all of the systems together physically
- 72. 72 API Failure • When an API fails, an organization can be exposed to high levels of risk • Often APIs reveal security, performance, reliability, portability, and many other types of risks • Risks accrue to both the organization providing an API and the organizations consuming it
- 73. 73 Barriers in API Testing • Experts note that the amount of time and resources needed to adequately test APIs is often the same as needed to analyze and develop them • This can cause problems with management who are used to much smaller test efforts relative to the development effort • API testing is black-box testing; and integration testing
- 74. 74 Use of Automation Tools for API Testing • The automation tools that are needed for API testing must have intuitive interfaces, automating as much of the test development as possible as well as the execution of the tests • The protocols used to define the APIs tend to be well developed and documented (if not fully mature) • That allows many of the tools in the market today to deliver facilities for creating multiple tests to cover the protocols • This helps the technical test analyst create more and better tests in less time
- 75. 75 Data Flow Testing • A white-box test design technique in which test cases are designed to execute definition and use pairs of variables • Many data flow issues are related to programming languages being used • Good technical test analyst should be able to investigate way data is used • No matter how good programmer; even great programmers generate bugs
- 76. 76
- 77. 77 Life Cycle of a Data Variable • These three atomic patterns are combined to show a data flow •d: Variable is created, defined, initialized, or changed •u: Variable may be used in a computation or in a decision predicate •k: Variable killed or destroyed, or the variable has gone out of scope • A ~ (tilde) is often used to show the first or last action that can occur
- 78. 78
- 79. 79
- 80. 80 Example: Cell-Phone Plan If the customer uses up to 100 minutes (inclusive), then there is a flat fee of $40.00 for the plan. For all minutes used from 101 to 200 minutes (inclusive), there is an added fee of $0.50 per minute. Any minutes after that used are billed at $0.10 per minute. Finally, if the bill is over $100.00, a 10 percent discount on the entire bill is given.
- 81. 81
- 82. 82 Immediate Question • A good tester might immediately ask the question as to how much is billed if the user does not use the cell phone at all • Our assumption—based on what we have seen in the real world— would be that the user would still have to pay the $40.00 • However, the code as given in Figure 3–6 lets the user off completely free if the phone was not used for the month
- 83. 83 $0.0 Bill • Line 3 looks at the number of minutes billed • If none were billed, the initial $40.00 is not billed; instead, it evaluates to FALSE and a $0.00 bill is sent • This as an error
- 84. 84 Bill Calculation • Bill is set to $40.00 in line 4 • In line 6 we check to see if there were more than 100 minutes used • In line 7 we check if more than 200 minutes were used • If not, we simply calculate the extra minutes over 100 and add $0.50 for each one • If over 200 minutes, we take the base bill, add $50.00 for the first extra 100 minutes, and then bill $0.10 per minute for all extra minutes • Finally, we calculate the discount if the bill is over $100.00
- 85. 85
- 86. 86
- 87. 87
- 88. 88
- 89. 89 Data Flow Patterns • Once the data flow patterns are defined, testing becomes a relatively simple case of selecting data such that each defined pair is covered • Of course, this does not guarantee that all data-related bugs will be uncovered via this testing
- 90. 90
- 91. 91
- 92. 92 â– One place where we have a first define (line 1) â– One potential bug where we double define (dd) in the flow 1-2-3 â– Seven separate du pairs where we define and then use Bill â– Three separate ud pairs where we use and then redefine Bill â– Two separate uu pairs where we use and then reuse Bill â– One uk pair where we use and then kill Bill â– And finally, one place where we kill Bill last
- 93. 93
- 94. 94
- 95. 95
- 96. 96 Reading • Related Topics from Ch 2 and Ch 3 from Advanced Software Testing 2/ed • Alastair Wilkes, Static Code Analysis Approaches for Handling Code Quality: How to Amplify Static Code Analysis Effectiveness by Layering on Predictive Test Selection. February 2, 2023 https://www.launchableinc.com/blog/static-code-analysis- approaches-for-handling-code-quality