Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017

Mutate and Test Your Tests
Author: Benoit Baudry
Project Coordinator and Scientific Leader
KTH, Sweden
1
baudry@kth.se caroline.landry@inria.fr
Speaker: Caroline Landry
Project Technical Manager
INRIA, France

POSS 17 - 2 Caroline Landry, INRIABenoit Baudry, KTH
Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs

Test Your Tests
•What do you expect from test cases?
• Cover requirements
• Stress the application
• Prevent regressions
• Reveal bugs

@Test
factorialWith5Test() {
long obs = fact(5);
assertTrue(5 < obs);
}

long fact(int n) {
if (n == 0)return 1;
long result = 1;
for(int i=2; i<=n; i++)
result = result * i;
return result;}
@Test
long obs = fact(5);
}

Coverage
long fact(int n) {
long result = 1;
return result;}
@Test
long obs = fact(5);
}

Coverage
@Test
long obs = fact(5);
}
@Test
assertEqual(1, fact(0));}
long fact(int n) {
long result = 1;
return result;}

@Test
long obs = fact(5);
}
@Test
long fact(int n) {
long result = 1;
return result;}
Are these test cases good at
detecting bugs?

@Test
long obs = fact(5);
}
@Test
long fact(int n) {
long result = 1;
return result;}
Are these test cases good at
detecting bugs?
Let’s mutate our code to see.

Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests

Mutation analysis
•Tests are good if they can detect bugs
•Principle: generate bugs and test the tests
•Mutation operators
•Mutant = Program with one mutation

Mutation analysis
Inputs : P, TS, Ops
Output : score, coverage
M <- generateMutants (P, OPs)
forAll (m in M)
run (TS,m)
if (one-test-fail)
then killed <- m
else alive <- m
score = killed / size(M)

PIT
•Open source
•Actively developed and maintained
•State of the art
•For Java
•Integrates with major build systems

PIT mutation operators
•Conditions
•Constants
•Return
•Delete method calls
•Constructor calls

long fact(int n) {
if(n == 0)return 1;
long result = 1;
return result;}

long fact(int n) {
if(n == 0)return 1;
long result = 1;
return result;}
n != 0

long fact(int n) {
if(n == 0)return 1;
long result = 1;
return result;}
n != 0 return 1+1

long fact(int n) {
if(n == 0)return 1;
long result = 1;
return result;}
n != 0 return 1+1
<

long fact(int n) {
if(n == 0)return 1;
long result = 1;
return result;}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

long fact(int n) {
long result = 1;
return result;}
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

long fact(int n) {
long result = 1;
return result;}
@Test
long obs = fact(5);
assertTrue(5 < obs); }
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

long fact(int n) {
long result = 1;
return result;}
@Test
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

long fact(int n) {
long result = 1;
return result;}
@Test
long obs = fact(5);
@Test
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

long fact(int n) {
long result = 1;
return result;}
Mutation reveals bugs in the test suite
@Test
long obs = fact(5);
@Test
Bugs in the test suite:
- Weak oracle
- Missing input
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

long fact(int n) {
long result = 1;
return result;}
@Test
long obs = fact(5);
assertEqual(120, obs); }
@Test
@Test
n != 0 return 1+1
< --!(i<=n)
result/i
result+1

PIT – reporting

Project #LOC #Mutants Time (h) Score (%)
Amazon Web Services SDK 1703892 2141690 04:25:35 76.28
XWiki Rendering Engine 104727 112609 01:59:55 50.89
Apache Commons Math 180951 104786 03:22:18 83.81
JFreeChart 134353 89592 00:41:28 58.04
Apache PdfBox 137099 79763 06:20:25 58.89
Java Git 128874 78316 16:02:00 73.86
SCIFIO 55347 62768 03:12:11 45.92
Joda-Time 85911 31233 00:16:32 81.65
Apache Commons Lang 60733 30361 00:21:02 86.17
Apache Commons Collections 51632 20394 00:05:41 85.94
Urban Airship Client Library 40885 17345 00:11:31 82.26
SAT4J 26415 17067 11:11:04 68.58
ImageJ Common 23868 15592 00:29:09 54.77
jsoup 17839 14054 00:12:49 78.34
Jaxen XPath Engine 21204 12210 00:24:40 67.13

Descartes
•Mutation operators: extreme mutation
•Active development
•Open source
•Compared to PIT
• Less mutants
• Different type of feedback
• Same framework

long fact(int n) {
long result = 1;
return result;}
Descartes - example

long fact(int n) {
long result = 1;
return result;}
Descartes - example
long fact(int n){
return 0;}
long fact(int n){
return 1;}
PIT : 7 mutants
Descartes : 2 mutants

Scalability of extreme mutation
public static boolean isValidXmlChar(int ch){
return (ch == 0x9)
|| (ch == 0xA)
|| (ch == 0xD)
|| (ch >= 0x20 && ch <= 0xD7FF)
|| (ch >= 0xE000 && ch <= 0xFFFD
|| (ch >= 0x10000 && ch <= 0x10FFFF);
}
PIT : 45 mutants
Descartes : 2 mutants

Project Mutants PIT Mutants Descartes Time PIT Time Descartes
Amazon Web Services SDK 2141690 161758 04:25:35 01:27:30
XWiki Rendering Engine 112609 5534 01:59:55 00:10:50
Apache Commons Math 104786 7150 03:22:18 00:08:30
JFreeChart 89592 7210 00:41:28 00:05:26
Apache PdfBox 79763 7559 06:20:25 00:42:11
Java Git 78316 7152 16:02:00 00:56:07
SCIFIO 62768 3627 03:12:11 00:15:26
Joda-Time 31233 4525 00:16:32 00:04:13
Apache Commons Lang 30361 3872 00:21:02 00:02:18
Apache Commons Collections 20394 3558 00:05:41 00:01:48
Urban Airship Client Library 17345 3082 00:11:31 00:09:38
SAT4J 17067 2296 11:11:04 00:56:42
ImageJ Common 15592 1947 00:29:09 00:04:08
jsoup 14054 1566 00:12:49 00:02:45

Coarser grain than PIT
long fact(int n) {
long result = 1;
return result;}
long fact(int n){
return 0;}
long fact(int n){
return 1;}
@Test
long obs = fact(5);
@Test

bool equals(object other) {
return other instanceof AClass
&&((AClass) other).aField==aField;
}
return true;}
return false;}

}
return true;}
return false;}
test() {
AClass a = new AClass(3);
AClass b = new AClass(3);
AClass c = new AClass(4);
assertEquals(a, b);
assertFalse(a == c);
}

}
return true;}
return false;}
test() {
assertEquals(a, b);
}

Wrong test
}
return true;}
return false;}
test() {
assertEquals(a, b);
}

protected final void prepareSocket(final SSLSocket socket) { }}

@Test
void typical() throws NoSuchAlgorithmException {
SdkTLSSocketFactory f = new SdkTLSSocketF(SSLContext.getDefault(),null);
f.prepareSocket(new TestSSLSocket() {
@Override
public void setEnabledProtocols(String[] protocols) {
assertTrue(Arrays.equals(protocols, new String[] {
"TLSv1.2", "TLSv1.1", "TLSv1", "SSLv3" }));
}});}}}

@Test
@Override
}});}}}

Missing oracle
@Test
@Override
}});}}}

private boolean isLongOption(final String token){
if (...){return false;} //Covered 27 times
...
if (...){return true;} //Not covered
else if (...){return true;} //Covered once
return false; //Covered 7 times
}

return false;}
return true;}

...
if (condition && isLongOption(value)){
action1(value);}
else{
action2(value);}
...
return false;}
return true;}

...
action1(value);}
else{
action2(value);}
...
return false;}
return true;}

...
action1(value);}
else{
action2(value);}
...
return false;}
return true;}
Testability issue

15
285
4 15 54 28 64 132 23
115
14
594 369
92 9 40
158
458 116
65
2592
69
2
14 28 19 27 55 10
17
12
97
85
31 2
14
11
61 20
69
197
274
1516
175 408 1178 610 1840 4245 459
344
585
2931 2237
2596 252 714
456 1726 425
2004
1618
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Tested
Weak pseudo-tested
Strong pseudo-tested

Descartes – future
•Incremental Descartes in the CI
•Github hook
•Support multi module Maven projects
•Enhanced diagnosis
•Integrate into a test augmentation pipeline

Conclusion
•Mutation analysis
• Automatic generation of mutants
• Evaluate the test suite
•Bugs in test suites
• Oracle
• Input space coverage
• Testability
• Indirectly tested code

Feedback welcome!
• https://github.com/STAMP-project/pitest-descartes
• https://github.com/hcoles/pitest
• http://stamp-project.eu/
baudry@kth.se
caroline.landry@inria.fr

Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017

More Related Content

What's hot (20)

Similar to Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017 (20)

More from STAMP Project (7)

Recently uploaded (20)

Mutate and Test your Tests with STAMP, Caroline Landry, Inria, Paris Open Source Summit 2017