Mutation Testing

Mutation Testing

Chris Sinjakli

Testing is a good thing

But how do we know our tests are
good?

Code coverage is a start

But it can give a “good” score with
really dreadful tests

Really dreadful tests
public int addTwoNumbers(int a, int b) {
return a – b;
}

...

@Test
public void shouldAddTwoNumbers() {
int result = addTwoNumbers(1, 1);
assertTrue(true);
}
Coverage: 100%
Usefulness: 0

A contrived example

But how could we detect it?

Mutation Testing!

“Who watches the watchmen?”

If you can change the code, and a
test doesn’t fail, either the code is
never run or the tests are wrong.

Going with our previous example
return a – b;
}
Let’s change something
...

@Test
assertTrue(true);
}

Going with our previous example
return a + b;
}

...
This still passes

@Test
assertTrue(true);
}

So it caught a really rubbish test

How about something slightly less
obvious?

Slightly less obvious (and I mean slightly)
public int checkConditions(boolean a, boolean b) {
if (a && b) {
return 42;
}
else {
return 0;
}
}

@Test
public void testBothFalse() {
int result = checkConditions(false, false);
assertEquals(0, result);
}
@Test
public void testBothTrue () {
int result = checkConditions(true, true);

} Coverage: 100%
Usefulness: >0
But still wrong

if (a && b) {
return 42;
} Mutate
else {
return 0;
}
}

@Test
public void testBothFalse() {
}
@Test

}

if (a || b) {
return 42;
}
else {
return 0;
}
}

@Test
public void testBothFalse() { Passing tests
}
@Test

}

Mutation testing caught our
mistake
:D

Useful technique

But still has its flaws

The downfall of mutation
(Equivalent Mutants)
int index = 0

while (someCondition) {
doStuff();
index++;
if (index == 100) {
break;
}
}
Mutates to
int index = 0

while (someCondition) {
doStuff();
index++;
if (index >= 100) {
break;
}
}

But the programs are equivalent, so no test will fail

Tools

Some Java, then some Ruby

Java
• Loads of tools to choose from
• Bytecode vs source mutation
• Will look at PIT (seems like one of the better
ones)

PIT - pitest.org
• Works with “everything”
– Command line
– Ant
– Maven
• Bytecode level mutations (faster)
• Very customisable
– Exclude classes/packages from mutation
– Choose which mutations you want
– Timeouts
• Makes pretty HTML reports (line/mutation coverage)

Ruby
• Mutant seems to be the new favourite
• Runs in Rubinius (1.8 or 1.9 mode)
• Only supports RSpec
• Easy to set up
rvm install rbx-head
rvm use rbx-head
gem install mutant
• And easy to use
mutate “ClassName#method_to_test” spec

Summary
• Seems like it could identify areas of weakness
in our tests
• At the same time, could be very noisy
• Might be worth just trying it against an
existing project and seeing what happens

Mutation Testing

More Related Content

What's hot (20)

Similar to Mutation Testing (20)

Recently uploaded (20)

Mutation Testing

Editor's Notes