froglogic Coco Code Coverage Presentation

Why Squish Coco?
● Typical question: Are we testing enough? Are we testing the right
things?
● Answer: Code coverage analysis
– Understand which tests excercise which source code
– Understand which tests are redundant
– Discover testing gaps
– More advanced analysis

Squish Coco
● Cross-Plattform Code Coverage Analysis
– Windows, Linux, macOS, Unix, RTOSes
– C, C++, C#, SystemC, Tcl, QML
● Coverage Levels
– Function
– Statement
– Branch (Decision)
– Condition
– Condition/Decision
– MC/DC
– MCC

Coco Agenda
Basics
●
Available coverage metrics
●
Supported platforms
●
Architecture, data formats
●
Approaches to instrumentation
●
Impact on build and execution
Live Demos
●
Interactive analysis via GUI
●
Report generation via command line
●
Overview of result formats
●
Manual validation
●
Analysis: optimzed execution order,
execution comparison, patch review
Integrations
●
Command line tools
●
Unit tests
●
CI systems
●
Build system
Discussion
●
Question and answers
●
Requirements

Coverage Levels – Function
int f(unsigned int a, unsigned int b, int c) {
int x = 0;
if ((a != 0 || b != 0) && c != 0)
x = a*b / c / max(a, b);
return x;
}
Test
f(0, 1, 1)

Coverage Levels – Line
int x = 0;
if ((a != 0 || b != 0) && c != 0)
x = a*b / c / max(a, b);
return x;
}
Test
f(0, 1, 1)

Coverage Levels – Statement
int x = 0;
if ((a != 0 || b != 0) && c != 0)
x = a*b / c / max(a, b);
return x;
}
Test
f(0, 1, 1)

Coverage Levels – Decision / Branch
int x = 0;
if ((a != 0 || b != 0) && c != 0)
x = a*b / c / max(a, b);
return x;
}
Test (X || Y) && Z Decision
f(0, 1, 1) (FALSE || TRUE) && TRUE TRUE
f(0, 0, 1) (FALSE || FALSE) && …. FALSE

Coverage Levels – Condition
int x = 0;
if ((a != 0 || b != 0) && c != 0)
x = a*b / c / max(a, b);
return x;
}
Test a != 0 b != 0 c != 0
f(1, 1, 0) TRUE - FALSE
f(0, 1, 1) FALSE TRUE TRUE
f(0, 0, 0) FALSE FALSE -

Coverage Levels – MCC
bool isValidPosition(int x, int y, int z)
{
if ((x > 10 || y > 20) && z > 0)
return true;
else
return false;
}
x > 10 y > 20 z > 0
FALSE TRUE TRUE
TRUE - TRUE
FALSE TRUE FALSE
TRUE - FALSE
FALSE FALSE -

Coverage Levels – MCC #2
bool isSilentSignal(int *line1, int *line2)
{
if ((!line1 || *line1 <= 0) && (!line2 || *line2 <= 0))
return true;
else
return false;
}
!line1 *line1 <= 0 !line2 *line2 <= 0
TRUE - FALSE TRUE
TRUE - FALSE FALSE
FALSE TRUE TRUE -
TRUE - TRUE -
FALSE FALSE - -
FALSE TRUE FALSE TRUE
FALSE TRUE FALSE FALSE

Modified Condition / Decision Coverage
“Every point of entry and exit in the program has been invoked at least once, every
condition in a decision in the program has taken all possible outcomes at least once, and
each condition has been shown to affect that decision outcome independently. A condition
is shown to affect a decision's outcome independently by varying just that condition while
holding fixed all other possible conditions.”
(Formal definition DO-178C)

Coverage Levels – MC/DC
bool isSilentSignal(int *line1, int *line2)
{
if ((!line1 || *line1 <= 0) && (!line2 || *line2 <= 0))
return true;
else
return false;
}
!line1 *line1 <= 0 !line2 *line2 <= 0 Decision
TRUE - FALSE TRUE TRUE
TRUE - FALSE FALSE FALSE
FALSE TRUE TRUE - TRUE
TRUE - TRUE - TRUE
FALSE FALSE - - FALSE

Supported Platforms
Operating systems
● Windows
●
Linux
●
Mac OS X
● Solaris
●
Embedded / RTOS
Programming languages
●
C and C++
●
SystemC
●
C# (Microsoft and Mono)
●
QML
●
Tcl
●
Planned: Java, JavaScript

Architecture
Instrumentation Database

Architecture
Report Generation

Architecture
Regular Product Build

Architecture
Regular Source Compilation

Architecture
Source Instrumentation

Architecture
Instrumentation Linking

Architecture
Alternative to build system change:
● Make use of fake cl.exe and link.exe
● Set PATH to override originals
● Set COVERAGESCANNER_ARGS=--cs-on

Instrumentation Types
●
Source code insertion at compile time
●
Pro: highest possible coverage type
●
Drawback: increased binary size
●
Binary instrumentation at runtime
●
Pro: No recompilation necessary
●
Con: Limited coverage type
●
Con: Less portable
●
Con: wrong results with optimized builds

Instrumentation Overhead
Impact on:
●
Compilation time
●
Binary size
●
Execution time
Influencing factors:
●
Coverage level
●
Optimization level

Report Generation
Command line version:
cmreport --title="Coverage Results"
--csmes=parser.exe.csmes
--select=".*"
--bargraph --toc
--global=all --method=all --source=all
--execution=all
--html=result.html
Additional options: HTML style, watermarks, coverage level, ...

Report Generation
Available formats:
●
HTML (stylable)
●
XML (basic, EMMA)
●
JUnit
●
CSV
●
Text (line format)
●
Cobertura (for SonarQube)

Manual Validation
How to deal with code hard/impossible to automatically cover?
●
Source code annotations
●
Pro: robust
●
Drawback: invasive
●
Example: /* coco validated: only for debugging */
●
Result amendments
●
Pro: product untouched
●
Con: possibly fragile on code changes

Command Line Tools
Execution import via cmcsexeimport:

Command Line Tools
Database merge via cmmerge:

Integration
System test with “dump on exit”:

Integration
Monitor long-running process with “dump on event”:

Integration
Force dump with API calls:

Integration
Control API (guard with #ifdef __COVERAGESCANNER__):
__coveragescanner_testname(const char *name)
__coveragescanner_teststate(const char *state)
__coveragescanner_add_html_comment(const char *comment)
__coveragescanner_save()
__coveragescanner_filename(const char *name)
__coveragescanner_set_custom_io(.....)
Pragmas:
#pragma CoverageScanner(cov-on)
#pragma CoverageScanner(cov-off)

Unit Test Integration
Basic integration
● Rely on “dump on exit”
● Make use of cmcsexeimport, cmmerge and cmreport
Advanced integration
● Set __coveragescanner_test_name()
● Transport __coveragescanner_test_state()
● Add __coveragescanner_html_comment()
● Segment with __coveragescanner_save()

Unit Test Integration (Qt)
Usage in QTestLib:
● Test build, execution and coverage as single step
●
Associate test run and result with respective coverage
...
void MyUnitTest::cleanup()
{
cleanupTest();
#ifdef __COVERAGESCANNER__
QString test_name="unittest/";
test_name += metaObject()->className();
test_name += "/";
test_name += QTest::currentTestFunction();
__coveragescanner_testname(test_name.toLatin1());
if (QTest::currentTestFailed())
__coveragescanner_teststate("FAILED");
else
__coveragescanner_teststate("PASSED") ;
__coveragescanner_save();
#endif
}

Test Framework Integration
●
Continuous Integration (CI):
●
Jenkins
●
Bamboo
●
CruiseControl
●
SonarQube
●
Hooks in revision control system

froglogic Coco Code Coverage Presentation

More Related Content

What's hot (20)

Similar to froglogic Coco Code Coverage Presentation (20)

Recently uploaded (20)

froglogic Coco Code Coverage Presentation