Mutation-Testing mit PIT

e-movimento Software Design & Beratung GmbH
1030 Wien ● Marxergasse 7/26 ► www.e-movimento.com
Mutation-Testing mit PIT
www.pitest.org

Mutation Testing – Was?
 Definition (https://en.wikipedia.org/wiki/Mutation_testing):
Mutation testing (or Mutation analysis or Program
mutation) is used to design new software tests and evaluate
the quality of existing software tests. Mutation testing
involves modifying a program in small ways. Each mutated
version is called a mutant and tests detect and reject
mutants by causing the behavior of the original version to
differ from the mutant. This is called killing the mutant. Test
suites are measured by the percentage of mutants that they
kill.
► www.e-movimento.com, Sebastian Dietrich2

Mutation Testing – Wozu?
 Ziel:
The purpose is to help the tester develop effective tests or
locate weaknesses in the test data used for the program or
in sections of the code that are seldom or never accessed
during execution. Mutation testing is a form of white-box
testing.
 Reicht das?
"Quis custodiet ipsos custodes?“
“Wer wird die Wächter selbst bewachen?“
[Juvenal, Satire 6, 347f.]

 muJava A mutation tool for Java that
includes class-level operators
 mutate.py A Python script to mutate
C-programs
 Mutator A source-based multi-
language commercial mutation
analyzer for concurrent Java, Ruby,
JavaScript and PHP
 Bacterio Mutation testing tool for
multi-class Java systems
 Javalanche Bytecode-based mutation
testing tool for Java
 Major Compiler-integrated mutation
testing framework for Java
 Jumble Bytecode-based mutation
 PIT Bytecode-based mutation
 Stryker Mutation testing tool for
JavaScript
 Mutant AST based mutation testing
tool for Ruby
 Jester Source-based mutation testing
tool for Java
 Judy Mutation testing tool for Java
 Heckle Mutation testing tool for Ruby
 NinjaTurtles IL-based mutation testing
tool for .NET and Mono
 Nester Mutation testing tool for C#
 Humbug Mutation testing tool for PHP
 MuCheck Mutation analysis library for
Haskell
Mutation Testing - PIT

PIT
 Ändert Bytecode und lässt
Tests laufen
 Arbeitet mit ant, maven,
gradle, ...
 Plugins für Eclipse, IDEA, ...
 Dunkelgrün = Mutation Coverage
 Hellgrün = Line Coverage
 Hellrot = keine Line Coverage
 Dunkelrot = keine Mutation Coverage

 Conditionals Boundary:
<  <=, >=  >, ...
 Negate Conditionals:
==  !=, >=  <
 Remove Conditionals:
a == b  true
 Math Mutator:
+  -, &  |, <<  >>, ...
 Increments Mutator:
i++  i—, i--  i++
 Invert Negatives:
-i  i
 Inline Constant:
int i = 5;  int i =-1;
 Return Values:
return true;  return false;
return x;  return x+1;
return obj;  return null;
return null;  RuntimeExc
 Void Method Call:
doSomething();  ;
 Non Void Method Call:
return5();  0;
 Constructor Call:
new Object();  null;
PIT – Mutators (Auszug – in Summe 7/13)

Praxis – UnitTests eines Moduls
 1.735 Tests  69,5% Coverage (lt. Sonar)
 Min. 5GB RAM für Maven-Build
  12857 Mutationen, davon 5312 gekillt (weil doppelt)
  19.117 Tests
  41% Mutation Coverage (63% Line Coverage)

Laufzeit
 mvn test  3:18 min.
vs.
 mvn pitest-maven:mutationCoverage 

Laufzeit – Aber...
 Incremental analysis möglich (nicht getestet):
 Mutation Killed, weder Test noch Klasse geändert  Mutation gilt
weiterhin als gekilled
 Mutation Überlebt, weder Test noch Klasse geändert, kein neuer
Test covered diese Klasse  Mutation gilt weiterhin als überlebt
 Mutation Killed, Klasse oder Test geändert  Mutation wird
vermutlich weiterhin gekilled werden  höhere Prio
 ...

Konklusio
 Für Coverage-Kennzahlen größerer Module ungeeignet
(da zu viel RAM nötig)
 Für Build vermutlich ungeeignet (da zu langsam –
incremental analysis check noch offen)
 Zum Schreiben/Perfektionieren/Verbessern einzelner
Unit-Tests gut geeignet
 Am besten mit Plugins arbeiten

Mutation-Testing mit PIT

More Related Content

What's hot (20)

Similar to Mutation-Testing mit PIT (20)

Recently uploaded (20)

Mutation-Testing mit PIT