ConFoo - Improve your tests with mutation testing

MUTATION TESTING
TO THE RESCUE OF
YOUR TESTS
NICOLAS FRÄNKEL

@nicolas_frankel #mutationtesting
2

HYBRIS, AN SAP COMPANY
3

ME, MYSELF AND I
By day
• Consultant
By night
• Teacher
• Book author
• Conference speaker
• Blogger
4

MANY KINDS OF TESTING
Unit Testing
Integration Testing
End-to-end Testing
Performance Testing
Penetration Testing
Exploratory Testing
etc.
5

THEIR ONLY SINGLE GOAL
Ensure the quality of the
production code
6

THE PROBLEM
How to check the quality of
the testing code?
7

CODE COVERAGE
“Code coverage is a
measure used to describe the
degree to which the source
code of a program is tested”
--Wikipedia
http://en.wikipedia.org/wiki/Co
de_coverage
8

MEASURING CODE COVERAGE
Check whether a source
code line is executed during a
test
• Or Branch Coverage
9

COMPUTING CODE COVERAGE
CC =
Lexecuted
Ltotal
*100
CC: Code Coverage
(in percent)
Lexecuted: Number of
executed lines of code
Ltotal: Number of total
lines of code
10

JAVA TOOLS FOR CODE COVERAGE
JaCoCo
Clover
Cobertura
etc.
11

100% CODE COVERAGE?
“Is 100% code coverage
realistic? Of course it is. If you
can write a line of code, you can
write another that tests it.”
Robert Martin (Uncle Bob)
https://twitter.com/unclebobmarti
n/status/55966620509667328
12

ASSERT-LESS TESTING
@Test
public void add_should_add() {
new Math().add(1, 1);
}
But, where is the
assert?
As long as the Code Coverage is
OK…
13

CODE COVERAGE AS A MEASURE OF QUALITY
Any metric can be gamed!
Code coverage is a metric…
⇒ Code coverage can be
gamed
• On purpose
• Or by accident
14

CODE COVERAGE AS A MEASURE OF QUALITY
Code Coverage lulls you into
a false sense of security…
15

THE PROBLEM STILL STANDS
Code coverage cannot
ensure test quality
• Is there another way?
Mutation Testing to the
rescue!
16

THE CAST
William Stryker
Original Source Code
Jason Stryker
Modified Source Code
a.k.a “The Mutant”
17

public class Math {
public int add(int i1, int i2) {
return i1 + i2;
}
}
public class Math {
return i1 - i2;
}
}
18

STANDARD TESTING
✔Execute Test
19

MUTATION TESTING
?Execute SAME Test
MUTATION
20

MUTATION TESTING
✗
✔Execute SAME Test
Execute SAME Test
Mutant
Killed
Mutant
Survived
21

KILLED OR SURVIVING?
Surviving means changing the
source code did not change the test
result
• It’s bad!
Killed means changing the source
code changed the test result
• It’s good
22

TEST THE CODE
✔Execute Test
public class Math {
return i1 - i2;
}
}
@Test
}
23

✔Execute Test
SURVIVING MUTANT
public class Math {
return i1 - i2;
}
}
@Test
}
24

TEST THE CODE
@Test
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
✔Execute Test
public class Math {
return i1 - i2;
}
}
25

KILLED MUTANT
✗Execute SAME Test
@Test
int sum = new Math().add(1, 1);
Assert.assertEquals(sum, 2);
}
public class Math {
return i1 - i2;
}
}
26

MUTATION TESTING IN JAVA
PIT is a tool for Mutation
testing
Available as
• Command-line tool
• Ant target
• Maven plugin
27

MUTATORS
Mutators are patterns
applied to source code to
produce mutations
28

PIT MUTATORS SAMPLE
Name Example source Result
Conditionals Boundary > >=
Negate Conditionals == !=
Remove Conditionals foo == bar true
Math + -
Increments foo++ foo--
Invert Negatives -foo foo
Inline Constant static final FOO= 42 static final FOO = 43
Return Values return true return false
Void Method Call System.out.println("foo")
Non Void Method Call long t = System.currentTimeMillis() long t = 0
Constructor Call Date d = new Date() Date d = null;
29

IMPORTANT MUTATORS
Conditionals Boundary
• Potential serious bug hiding
there
• if (foo > bar)
30

IMPORTANT MUTATORS
 Void Method Call
• Assert.checkNotNull()
• connection.close()
31

REMEMBER
 It’s not because the IDE
generates code safely that
it will never change
• equals()
• hashCode()
32

ENOUGH TALK…
33

SH… HAPPENS
 False positives
 Imperfect
 Sloooooooooooooooow
34

PIT IS IMPERFECT
if (p < 0)
...
// changed condition boundary -> survived:
if (p > 0)
...
return 0;
35

PIT IS IMPERFECT
void rebootMachine() {
// removed method call:
checkUserPermissions();
Runtime.getRuntime().exec("reboot");
}
36

WHY SO SLOW?
Analyze test code
For each class under test
• For each mutator
• Create mutation
• For each mutation
• Run test
• Analyze result
• Aggregate results
37

WORKAROUNDS
Increase number of threads  
Set a limited a set of mutators
Limit scope of target classes
Limit number of tests
Limit dependency distance
Use incremental analysis  
Don’t bind to the test phase  
Use scmMutationCoverage
38

INCREMENTAL ANALYSIS
 Metadata stored between runs
 Mutant will not be checked again, if:
• killed, and neither class nor test have changed
• survived, and there are no new/changed tests for it
39

THE SILVER BULLET?
Checks the relevance of
your unit tests
Points out potential bugs
40

THE REST IS UP TO YOU
Validate the assembled application
• Integration Testing
Check the performance
• Performance Testing
Look out for display bugs
• End-to-end testing
Etc.
41

TESTING IS ABOUT ROI
Don’t test to achieve 100%
coverage
Test because it saves money
in the long run
Prioritize:
• Business-critical code
• Complex code
42

Q&A
http://blog.frankel.ch/
@nicolas_frankel
https://git.io/vznQK
43

ConFoo - Improve your tests with mutation testing

More Related Content

Similar to ConFoo - Improve your tests with mutation testing (10)

More from Nicolas Fränkel (20)

Recently uploaded (20)

ConFoo - Improve your tests with mutation testing