Code coverage in theory and in practice form the do178 b perspective
Download as PPT, PDF1 like1,049 views
The document discusses code coverage in the context of DO-178B, a standard for safety-critical software in aviation. It outlines five levels of software criticality, detailing the testing requirements for each level, with level A being the most critical and requiring the most extensive testing. It emphasizes the importance of code coverage analysis and its metrics, including method, statement, branch, condition, and path coverage, and highlights the complexities and challenges associated with achieving high coverage rates.
![Condition Coverage Condition coverage reports the true or false outcome of each condition. A condition is an operand of a logical operator that does not contain logical operators. Condition coverage measures the conditions independently of each other. This metric is similar to decision coverage but has better sensitivity to the control flow. However, full condition coverage does not guarantee full decision coverage . For example, consider the following C++/Java fragment. bool f(bool e) { return false; } bool a[2] = { false, false }; if (f(a && b)) ... if (a[int(a && b)]) ... if ((a && b) ? false : false) ... All three of the if-statements above branch false regardless of the values of a and b. However if you exercise this code with a and b having all possible combinations of values, condition coverage reports full coverage.](https://image.slidesharecdn.com/codecoverageintheoryandinpracticeformthedo178bperspective-111202084500-phpapp02/85/Code-coverage-in-theory-and-in-practice-form-the-do178-b-perspective-26-320.jpg)
![How It is Done? - Automatic Instrumentation. Instrumentation for statement coverage 1 char inst[5]; 2 void foo() 3 { 4 found= false ; 5 for (i=0;(i<100) && (! found);i++) 6 { 7 if (i==50 ) { inst[0]=1 ; break ;} 8 if (i==20 ) { inst[1]=1 ;found= true ;} 9 if (i==30 ) { inst[2]=1 ;found= true ;} 10 inst[3]=1 ; } 11 printf("foo\n"); 12 inst[4]=1 ; }](https://image.slidesharecdn.com/codecoverageintheoryandinpracticeformthedo178bperspective-111202084500-phpapp02/85/Code-coverage-in-theory-and-in-practice-form-the-do178-b-perspective-33-320.jpg)
![How It is Done? - Instrumentation Inserting the full instrumentation code for the condition coverage in this example will produce the following code: 1 char inst[15] ; 2 void foo() 3 { 4 found= false ; 5 for (i=0;((i<100)? inst[0]=1 :inst[1]=1,0) && ((! found)? inst[2]=1 :inst[3]=1,0);i++) 6 { 7 if ((i==50? inst[4]=1 : inst[5]=1 ,0) ) { inst[6]=1 ; break ;} 8 if ((i==20? inst[7]=1 : inst[8]=1 ,0) ) { inst[9]=1 ;found= true ;} 9 if ((i==30? inst[10]=1 : inst[11]=1 ,0) ) { inst[12]=1 ;found= true ;} 10 inst[13]=1 ; } 11 printf("foo\n"); 12 inst[14]=1 ; } Full code coverage instrumentation at condition level](https://image.slidesharecdn.com/codecoverageintheoryandinpracticeformthedo178bperspective-111202084500-phpapp02/85/Code-coverage-in-theory-and-in-practice-form-the-do178-b-perspective-34-320.jpg)
![Thank You Daniel Liezrowice – Engineering Software Lab (The Israeli Center for Static Code Analysis & Dynamic testing) [email_address] 09-8855803 www.eswlab.com](https://image.slidesharecdn.com/codecoverageintheoryandinpracticeformthedo178bperspective-111202084500-phpapp02/85/Code-coverage-in-theory-and-in-practice-form-the-do178-b-perspective-37-320.jpg)
Code coverage in theory and in practice form the do178 b perspective
- 1. Code Coverage in Theory and in practice form the DO178B perspective Daniel Liezrowice – Engineering Software Lab (The Israeli Center for Static Code Analysis ) Presentation is Supported by Parasoft Makers of Automated DO178B Certified Analysis Tools & A Wind River Certified partner
- 2. Testing of Code Coverage is essential part of Safety-Critical Software Certification
- 3. A little bit of Baselining Why Certify? What is DO-178B? DO-178B Criticality Levels Effect of Criticality Levels on Testing
- 4. Why Certify? Software flying on a commercial aircraft and sometimes military aircraft must adhere to DO-178B. Meant to ensure software safety, provide high software quality, and high reliability.
- 5. What is DO-178B? Developed by the International Commercial Aviation Community. Technically an established set of guidelines for developing safety-critical software. In reality, it’s a set of requirements for safety-critical software development. Part of DO-178B identifies “levels” of software criticality and what is required to certify each of the different levels.
- 6. Software Criticality Levels DO-178B categorizes five levels of software criticality Levels A through E, with Level A being the highest level of criticality. Application of DO-178B to Level E systems is not required
- 7. Software Criticality Levels Level A: anomalous software behavior would cause or contribute to a catastrophic failure condition resulting in a total loss of life. Level B: anomalous software behavior would cause or contribute to a hazardous/sever-major failure condition, resulting in some loss of life.
- 8. Software Criticality Levels Level C: anomalous behavior would cause or contribute to a major failure condition, resulting in serious injuries . Level D: anomalous behavior would cause or contribute to a minor failure condition, resulting in minor injuries . Level E: anomalous behavior would not effect aircraft operation or pilot workload, resulting in no impact on passenger or aircraft safety .
- 9. Testing Criticality Levels Level D Must have some verification process in place, but no structural code testing is required. Level C Must perform everything required by Level D Additionally, must perform “Statement Coverage” Statement Coverage - Every statement of source code must be executed by a formal test case
- 10. Testing Criticality Levels Level B Must perform everything required by Level C Additionally must perform “Decision/Condition” coverage Decision/Condition Coverage - Every code branch must be executed by a formal test case.
- 11. Testing Criticality Levels Example: int* p = NULL; if (condition) p = &variable; *p = 123;
- 12. Testing Criticality Levels Level A Must perform everything required by Level B Additionally must perform “Modified Condition/Decision Coverage” Modified Condition/Decision Coverage - Every condition within each decision statement needs to be independently verified for its effect on the statement.
- 13. Testing Criticality Levels Example: if ((A || B) && C) // Do something else // Do something else
- 14. Sub summery Avionics software must adhere to the DO-178B guidelines before it can fly. The DO-178B guidelines specify five levels of software criticality. The level of testing required varies by the software criticality level. The structural coverage testing required by DO-178B is complex and is a primary cost driver on avionics software projects .
- 15. Code Coverage Agenda Introduction to structural test case investigation techniques Why do we measure code coverage? Method Coverage Statement Coverage Branch Coverage Condition Coverage Path Coverage Limitations of structural techniques and coverage measurement
- 16. Code Coverage classified as White Box Testing The basis for designing white box tests is the source code of the software under test Program source must be available Basic idea: Try to execute all parts of the source at least once
- 17. Working with Code Coverage Code coverage analysis is the process of: Writing test cases and execute them Finding areas of code not covered by a set of test cases. Creating additional test cases to increase coverage. Determining a quantitative measure of code coverage Code Coverage is an indirect measure of quality. A code coverage analyser tool supports and automates this process. Code coverage is not fool-proof There is still the possibility of errors, even with 100% code coverage
- 18. Making Good Use of Code Coverage Improve tests using code coverage Design functional unit tests by using black-box tes t design techniques based on the Interface of the component. Measure code coverage and visualize the result to identify missing tests. Add some functional unit tests by using white-box test design techniques to reach the expected/target code coverage. Code Coverage Trend Reports Code Coverage Trend Reports are an important indicator for the health of your project. Beware of code coverage as acceptance criteria Coverage above 90% is incredibly difficult to achieve. If a certain level of code coverage is requested, developers tend to concentrate on easy unit tests, missing the hard and important ones. It may make sense to define hard coverage criteria for more complex components, as a result of a risk analysis, or if regulations exist (e.g. FDA for medical devices).
- 19. Code Coverage Tools Visualization of code not covered in the code editor Visualization of coverage rate achieved Warnings generated by the tool to point to code not covered Coverage rate achieved broken down by packages and classes
- 20. Method Coverage (Function Coverage) Reports whether a method (function) was invoked while testing the application. Identifies untested classes/methods. Is the weakest of all code coverage models. May help identifying where to focus testing at the beginning of a testing phase. No meaningful statement about the quality of test (coverage) possible.
- 21. Statement Coverage (Line Coverage) Reports whether each executable statement was executed. Identifies statements that are never called (Dead Code) High statement coverage is still far from good test coverage. Lot’s of bugs may still be undetected! true covers all statement
- 22. Statement Coverage (Line Coverage) The main disadvantage of statement coverage is that it is insensitive to some control structures int* p = NULL; if ( condition ) p = &variable; *p = 123; Without a test case that causes condition to evaluate false, statement coverage rates this code fully covered. In fact, if condition ever evaluates false, this code fails.
- 23. Branch Coverage (Decision Coverage) Reports whether boolean expressions evaluate to true AND false. Ignores branches within boolean expressions which occur due to short-circuit operators High branch coverage shows that some substantial unit testing was done May still miss errors in complex (compound) boolean expressions because of its focus of the two possible outcomes (true or false). true false
- 24. Branch Coverage (Decision Coverage) This metric has the advantage of simplicity without the problems of statement coverage . A disadvantage is that this metric ignores branches within Boolean expressions which occur due to short-circuit operators. For example if (condition1 && (condition2 || function1())) statement1; else statement2; This metric could consider the control structure completely exercised without a call to function1. The test expression is true when condition1 is true and condition2 is true, and the test expression is false when condition1 is false. In this instance, the short-circuit operators preclude a call to function1.
- 25. Condition Coverage Reports the true or false outcome of each boolean sub-expression similar to branch coverage but has better sensitivity to the control flow But (simple) condition coverage does not guarantee full branch coverage. Variations (Simple) Condition Coverage Multiple Condition Coverage ... true, true true, false false, true false, false
- 26. Condition Coverage Condition coverage reports the true or false outcome of each condition. A condition is an operand of a logical operator that does not contain logical operators. Condition coverage measures the conditions independently of each other. This metric is similar to decision coverage but has better sensitivity to the control flow. However, full condition coverage does not guarantee full decision coverage . For example, consider the following C++/Java fragment. bool f(bool e) { return false; } bool a[2] = { false, false }; if (f(a && b)) ... if (a[int(a && b)]) ... if ((a && b) ? false : false) ... All three of the if-statements above branch false regardless of the values of a and b. However if you exercise this code with a and b having all possible combinations of values, condition coverage reports full coverage.
- 27. Path Coverage (Predicate Coverage) Reports whether each of the possible paths in each method have been followed A path is a unique sequence of branches from the method entry to the exit. 100% Path coverage may not be achieved because: Many paths are impossible to exercise (dynamically) due to relationships of data. The number of paths is exponential to the number of branches. (in some cases we can even reach the “halting problem") Considers only a limited number of looping possibilities. Variations of Path Coverage data flow coverage (By Static code Analysis)
- 28. False Positive Problem: False Errors Solution By Data Flow Analysis false error : reported by analyzer but not in fact a latent error in the program 10-Mar-05 1 int f(int x) { 2 int y; 3 if (x > 0) y = x; 4 if (x > 3) x = y; 5 return x; 6 } 3 4 6 (x ≤ 0) (x > 0) y = x 5 (x ≤ 3) (x > 3) x = y return x Warning Variable 'y' (line 2) may not have been initialized (x ≤ 0) (x > 3)
- 29. Code Coverage: Brief Summary Method coverage Statement coverage Branch coverage Condition coverage Path coverage
- 30. Condition/Decision Coverage Condition/Decision Coverage is a hybrid metric composed by the union of condition coverage and decision coverage .
- 31. Modified Condition/Decision Coverage DO178B Level A Every point of entry and exit in the program has been invoked at least once, every condition in a decision has taken all possible outcomes at least once, every decision in the program has taken all possible outcomes at least once, and each condition in a decision has been shown to independently affect that decisions outcome. A condition is shown to independently affect a decisions outcome by varying just that condition while holding fixed all other possible conditions This metric is specified for safety critical aviation software by RCTA/DO-178B
- 32. How It is Done? - Instrumentation 1 void foo() 2 { 3 found= false ; 4 for (i=0;(i<100) && ( ! found );i++) 5 { 6 if (i==50) break ; 7 if (i==20) found= true ; 8 if (i==30) found= true ; 9 } 10 printf("foo\n"); 11 }
- 33. How It is Done? - Automatic Instrumentation. Instrumentation for statement coverage 1 char inst[5]; 2 void foo() 3 { 4 found= false ; 5 for (i=0;(i<100) && (! found);i++) 6 { 7 if (i==50 ) { inst[0]=1 ; break ;} 8 if (i==20 ) { inst[1]=1 ;found= true ;} 9 if (i==30 ) { inst[2]=1 ;found= true ;} 10 inst[3]=1 ; } 11 printf("foo\n"); 12 inst[4]=1 ; }
- 34. How It is Done? - Instrumentation Inserting the full instrumentation code for the condition coverage in this example will produce the following code: 1 char inst[15] ; 2 void foo() 3 { 4 found= false ; 5 for (i=0;((i<100)? inst[0]=1 :inst[1]=1,0) && ((! found)? inst[2]=1 :inst[3]=1,0);i++) 6 { 7 if ((i==50? inst[4]=1 : inst[5]=1 ,0) ) { inst[6]=1 ; break ;} 8 if ((i==20? inst[7]=1 : inst[8]=1 ,0) ) { inst[9]=1 ;found= true ;} 9 if ((i==30? inst[10]=1 : inst[11]=1 ,0) ) { inst[12]=1 ;found= true ;} 10 inst[13]=1 ; } 11 printf("foo\n"); 12 inst[14]=1 ; } Full code coverage instrumentation at condition level
- 35. References Cornett, Steve. “Code Coverage Analysis” <http://www.bullseye.com/coverage.html#basic_statement> Enea Teksci. “DO-178B Training”. March 2007 High Rely, Reliable Embedded Solutions. “FAA Compliance Questions and Answers” <http://www.highrely.com/do178b_questions.php> Nilsen, Kelvin. “Certification Requirements for Safety-Critical Software” <http://www.rtcmagazine.com/home/article.php?id=100010&pg=1>
- 36. We now moving to the Demo but before… Questions?
- 37. Thank You Daniel Liezrowice – Engineering Software Lab (The Israeli Center for Static Code Analysis & Dynamic testing) [email_address] 09-8855803 www.eswlab.com
Editor's Notes
- #12: Statement Coverage only requires one test case (condition evaluates to true) to satisfy this code Decision/Condition Coverage requires two test cases (condition evaluates to true and false) to satisfy this code
- #14: Decision/Condition Coverage only requires two test cases, one to execute the if branch, one to execute the else branch MCDC requires 4 test cases Must make “A OR B” both true and false, while simultaneously making the entire statement true and false.
- #17: White box testing is a structural technique, which is based on the structure of the test item. This structure is evaluated and measurements are taken from it. These assess characteristics such as The control flow of the item under test Data flows into, out of and within the item under test Which part of the source that have been covered by a particular set of tests In order to take these measurements, it is necessary to enhance the code with additional statements that record key events during execution. This is called Instrumentation As instrumentation may influence the behaviour of the test object, all tests should be repeated without coverage measurement, to exclude the possibility of side effects.
- #19: Code coverage as acceptance criteria: Statement about expected coverage in relation to the maximal possible coverage in terms of a certain coverage measurement. E.g. 90 % statement coverage means: 90% of all statements in the source code are covered by the tests executed. Empirical studies have shown that a incredibly difficult to achieve coverage rate must be reached to improve error detection noticeably (90 – 95 % or higher). This implies that punctually using coverage measurement to visualize areas of the code that are not covered with tests is superior to acceptance criteria using coverage measurement. Code Coverage as acceptance criteria may motivate developers to concentrate on developing test code to increase coverage (e.g. testing getter and setters) instead of to find errors in the code. It makes sense to concentrate on critical and complex parts of the code, where most bugs are expected to be. But those parts are often also hard to test and code coverage is increased only to a negligible degree by these tests.
- #20: A good code coverage measurement tool is not only capable of measuring code coverage and check whether some acceptance criteria in terms of code coverage are met. A good code coverage tool should also provide means to visualize achieved coverage in order to help developers identifying source code in need of additional tests. A good example is the reasonable priced Clover, a coverage measurement and visualization tool for Java available from www.cenqua.com. It integrates nicely with the Eclipse IDE and will be used in the labs for this module.
- #21: Method coverage doesn’t really help a lot. There is a need for more elaborate code coverage models.
- #22: While statement coverage certainly is a weak coverage measurement, it can make sense to use it, if the code itself is not complex and does not contain many branches.
- #26: Condition coverage reports the true or false outcome of each boolean sub-expression, separated by logical-and and logical-or if they occur. Condition coverage measures the sub-expressions independently of each other. Multiple condition coverage reports whether every possible combination of boolean sub-expressions occurs. Condition/Decision Coverage is a hybrid measure composed by the union of condition coverage and decision coverage.
- #29: following red path, warning seems justified but red path is infeasible given error is from Flexelint; Orion reports no errors
- #30: Method coverage focuses on called methods. Statement coverage focuses on called statements. Branch coverage focuses on decisions based on results of boolean expressions (true, false). Condition coverage focuses on decisions based each condition of an boolean expression. Path coverage focuses on each possible unique path through a method. Test coverage focuses on finding and elimination errors in the code.