MASTER-CLASS: "CODE COVERAGE ON Μ-CONTROLLER" Sebastian Götzinger

www.verifysoft.com
Code coverage for small
safety-critical embedded
targets

www.verifysoft.com 2
Agenda
1. Speaker Introduction
2. Verifysoft Short Introduction
3. Why Code Coverage?
4. Safety Standards and Code Coverage
5. Different Coverage Levels
6. Compiler Support
7. How does it work? Code Instrumention
8. Support for Embedded Targets
9. Qualification Kit
10. Different Reports
11. Supported Platforms/IDEand Tool Integrations
12. Live Demo

1. Introduction – Meet the speaker
Sebastian Götzinger
• ISTQB® Certified Tester Foundation Level
• Employed since 2013 at
Verifysoft Technology GmbH
• Trainer for test tools

2. Where does he come from?
VerifysoftTechnologyGmbH
• Founded 2003
• Headquarters: Offenburg, Germany
• >400 customers in more than 30 countries
• Owner of:
▪ Testwell CTC++ (Code Coverage)
▪ Testwell CMT++ (SW-Metrics)
▪ Testwell CTA++ (Unit-Tests)
• Distrubution of complementary Software
(e.g. Grammatec Codesonar® in Germany)
• Seminars for software quality

Code Coverage
shows the parts of the code
which have been executed / not executed
tested / not tested
Mandatory for security-critical development
-> security standards
Code Coverage Test Coverage Requirement Coverage

Why measure Code Coverage ?
• Helps to write better (more adapted) tests
• Helps to avoid redundant tests
• You know when you can stop testing
• Prove to your customers you have good quality
• You can be sure that your development partner delivers
good quality
• Helps to find Dead Code
• Required to obtain certifications
• Mandatory for security critical development
(standards DO-178C, IEC 61508, EN 50128, ISO 26262, …)

Code Coverage
shows the parts of the code
which have been executed / not executed
tested / not tested
Mandatory for security-critical development
-> security standards

Table B.2 from DIN EN 61508-3
++ Highly recommended
+ Recommended
DIN EN 61508-3

++ Highly recommended
+ Recommended
ISO 26262-6

int goo( int a, int b, int c)
{
int x;
if (((a>0) || (b>0)) && (c>0))
{
x = 1;
}
else
{
x = 0;
}
return x;
}
DO-178B Definition:
„Every decision has taken all possible outcomes at least once, and every
condition in a decision is shown to independently affect that decision‘s
outcome.“
(a>0) (b>0) (c>0) x
true - true 1
true - false 0
false true true 1
false true false 0
false false - 0
Incomplete Evaluation in C/C++
- is not to be evaluated, can be true or false
n+1 Tests necessary, n := amount of atomic conditions
12

Code Coverage Requirements: CENELEC EN 50128

5. Different Coverage Levels
Testwell CTC++ supports all required coverage levels:
• Function Coverage
• Decision Coverage
• Statement Coverage
• Condition Coverage
• Modified Condition/Decision Coverage (MC/DC)
• Multicondition Coverage (MCC)
works together with all unit-test tools

6. Compiler Support
Testwell CTC++ works with all compilers
Support is available for (as of March 2014, for actual list refer to
www.verifysoft.com/en_compilers.html):
Altium Tasking classic toolsets,VX-toolsettoolsets,c166,cc166, ccm16c,cc51
ARM DS-5, Keil MDK-ARM
Borland/Inprise/Paradigm/Codegear bcc, bcc32,pcc, pcc32 (Paradigm)
Ceva DSP all (justusegcc settings)
Cosmic cx6805,cx6808,cx6812, cxs12x,cxs12z, cxxgate, cx6811,cx6816,cx332,cxst10,cxstm8, cxst7,
cxcf,cx56K, cxppc
Freescale/Metrowerks mwccmcf, mwcceppc, mwccmcore, mwcc56800, mwcc56800e, chc12, chc08
Fujitsu/Softune fcc907s,fcc911s,fcc896s
gcc and all gcc based cross-compilers i586-mingw32msvc-gcc,x86_64-linux-gnu-gcc,m68k-palmos-coff-
gcc, tricore-gcc,arm-linux-gnueabi-gcc,arm-none-eabi-gcc,arm-none-linux-gnueabi-gcc,arm-elf-gcc,
arm-montavista-linux-gnueabi-gcc,pic30-gcc,pic32-gcc,avr-gcc,xc16-gcc,mlx16-gcc,thumb-epoc-pe-
gcc, arm4-epoc-pe-gcc, armv-epoc-pe-gcc, powerpc-wrs-linux-gnu-e500v2-glibc_small-gcc,*-gcc,*-*-gcc,
*-*-*-gcc
GHS/GreenHills/Multi ccv850,cxv850, ccmips,cxmips,ccarm,cxarm,ccthumb, cxthumb, ccppc,
cxppc,gcc (GreenHill,not GNU)
Hitatchi shc, shcpp,ch38,cxrx
HI-Tech PICC (Windows and Linux) picc,picc18,picc32, dspicc,xc16-gcc,xc32-gcc,
HP HPUX CC, HP C++, aCC
IAR compilers and toolchains icc430,icc78k,icc78k0r,icc8051, iccarm,iccavr,iccavr32,icccf,icchcs12,
iccdspic,iccmaxq,iccpic18,icccr16c,iccv850,icch8,iccm8k,iccm32c,iccm16c,iccr32c,iccrl78, iccrx,
iccsam8,iccstm8
Intel (all platforms) icc,ic86,ic96
Java compilers Javac,jikes,ecj,gcj,kaffe
Keil c51,c166, c251,ca/ cx51, cx2, tcc / armcc
LLVM clang,clang++/ Matlab/Simulink / lcc
Metaware (both Windows and Linux host) hcac,hcarc,hcarm
Microchip MPLAB C pic30-gcc,pic32-gcc
---->

6. Compiler Support
Testwell CTC++ works with all compilers (continued)
→
Microsoft compilers cl on host, both 32 and 64 bit / cl for Smartphones and PocketPC / csc C# compiler
/ vjc J# compiler
Mono compilers dmcs, mcs, gmcs, smcs
Motorola chc12, chc08
Pathscale pathcc/pathCC
Renesas shc,shcpp,ch38, ccrx
Raisonance rc51,rcmp
Sun WorkShop compilers,javac
Symbian various compilers
TI Code Composer Studio (Windows) cl2000,cl16x,cl470,cl55,cl500,cl430
Texas Instruments Linux compilers cl2000,cl16x,cl470,cl55,cl500,cl430
Trimedia tmcc
VisualDSP++ ccblkfn,cc21k,ccts
Windriver ccarm, ccsimpc,g++simpc,g++arm, cchppa,ccsimso,ccsparc,cc68k,cc386,cc960, ccmips,
ccppc
You have not seen your compiler? Contactus!
We will adapt Testwell CTC++ to your compiler within a
few daysand without any cost (adaptationcan even be
done by the customer).
Testwell CTC++ supports all compilers!
No unsupportedcompilers!

Instrumentation
• Adding of global counters (Integer-Arrays)
into the source code
• Storage of information about counter instrumentation
• Increment counters with each run
• Storage of counter values
• Analysis of the counter values for reporting
17
7. Code Instrumentation

Pre-
processor
Compiler Linker
Testbed
*.c *.i *.o *.11
Tests
Protocol
Testcoverage Tool
RAM
ROM
Memory requirement
without instrumentation
18
Tool-Chain

Pre-
processor
Compiler Linker
Testbed
*.c *.i *.o *.11
Tests
Protocol
Instrumen-
tation
Report
Generator
*.c
Coverage Report
Symbol data
Datafile
(counter)
Testcoverage Tool
RAM
ROM
Memory requirement
without instrumentation
RAM
ROM
Memory requirement
with instrumentation
19
Tool-Chain

• RAM
Reason for lack of memory: 80 % RAM, 20 % ROM (pract. experience)
• ROM
• Mostly no filesystem (so counters have to be stored in memory)
• Limited amount of interfaces on the target device (transfer of datafile)
Consider additional testing interfaces in the hardware design
(design for test)

Testwell CTC++ has a very low
instrumentationoverhead
It works with all embedded targets
If instrumentationoverheadit to high
We can use Bit Cov Add-on
Thus Testwell CTC++ works with even smallest
targets and microcontrollers

ROM-Usage
Without instrumentation: 60 Byte
Function Coverage: 67 Byte
Branch Coverage: 118 Byte
Condition Coverage: 285 Byte
Simple example with small code and big
instrumentation overhead (mean 30 % of code size).
Additional RAM-Usage without Bit-Coverage
Function Coverage: 1 Integer
Branch Coverage: 4 Integer
ConditionCoverage: 7 Integer
Additional RAM-Usage using Bit-Coverage
Function Coverage: 1 Bit
Branch Coverage: 4 Bit
ConditionCoverage: 7 Bit
Integer:
32 Bit (unsigned long)
as default

Testwell CTC++ is the ideal tool for embedded targets
• Dramatically easy to use !
• Very low instrumentationoverhead on your C files
• Works with all targets
Host-Target add-on is provided as source code
and so can be easily adapted to new targets
• … even with smallest targets and microcontrollers
• Supports all compilers/cross-compilers
8. Support for Embedded Targets and native projects

Testwell CTC++ is the ideal tool for native projects
• Setup and usage are straightforward
• Java and C# support on top of C and C++
• Very fast analysis
• Interfacing with MS Visual Studio IDE (and other IDEs)
• … even on large projects
8. Support for Embedded Targets and native projects

9. Qualification Kit
Testwell CTC++ can be used to obtain certification
in automotive, railway, avionics and medical
industries
Tool-Qualification Kits available

Reports in text, XML,HTML, JSON
Directory Summary
Files Summary
Functions Summary
Execution Profile
Untested Code Listing
Execution Time Listing

11. Supported Platforms/IDE and Tool Integrations
Supported Platforms
Windows, Linux, FreeBSD, Solaris, HP-UX
MacOSX, AIX, others on request

IDE-Integrations
Visual Studio v7.0 and later
IAR (all platforms)
Borland 5.02
Fujitsu Softune
Eclipse
Others on request
Works also with:
MP-LAB, Keil, …

Integrations
with Tool-Chains and Testing environments
include
CATIA - AUTOSAR Builder (DassaultSystèmes)
dSpace SystemDesk
dSpace TargetLink
PikeTech TPT
QTronic TestWeaver / Silver
Imagix 4D
SonarQube
… (ask for other integrations)
Further information: www.verifysoft.com

12. Live Demo
For a live demo, please visit OUR BOOTH 95 or:
http://www.verifysoft.com/en_ctcpp_online_presentations.html

Testwell CTC++ Test Coverage Analyser for C and C++
CTC++ add-on for Java and Android
CTC++ add-on for C#
Testwell CTC++
All coverage levels
Statement coverage
Function coverage
Decision/branch coverage
Condition coverage
Modified condition/decision cov.
MC/DC coverage
Multicondition coverage (MCC)
All compilers
All embedded targets!
Works with all unit test tools

Customers

Thank you for your time!
Visit our BOOTH 95!
Your Verifysoft Team
Thank You

MASTER-CLASS: "CODE COVERAGE ON Μ-CONTROLLER" Sebastian Götzinger

More Related Content

What's hot (20)

Similar to MASTER-CLASS: "CODE COVERAGE ON Μ-CONTROLLER" Sebastian Götzinger (20)

More from Ievgenii Katsan (20)

Recently uploaded (20)

MASTER-CLASS: "CODE COVERAGE ON Μ-CONTROLLER" Sebastian Götzinger