Mutation testing with PIT
- 1. Mutation Testing with PIT Rafał Leszko @RafalLeszko rafalleszko.com Hazelcast
- 2. About me ● Integration Team Lead at Hazelcast ● Worked at Google and CERN ● Author of the book "Continuous Delivery with Docker and Jenkins" ● Trainer and conference speaker ● Live in Kraków, Poland
- 3. About Hazelcast ● Distributed Company ● Open Source Software ● 140+ Employees ● Products: ○ Hazelcast IMDG ○ Hazelcast Jet ○ Hazelcast Cloud @Hazelcast
- 6. I wrote code for the spacecraft.
- 7. How do you that it works?
- 8. Hmm...
- 9. I just know
- 10. I feel it
- 11. I feel it
- 12. I feel it
- 13. I feel it
- 16. $327.6 million
- 23. I wrote code for the spacecraft.
- 24. How do you that it works?
- 25. I wrote unit tests!
- 26. How do you know that your tests work?
- 27. Hmm...
- 28. really?
- 29. code
- 30. code test
- 31. code test test
- 35. Does testing make any sense at all?
- 36. return a + b;
- 37. return a + b;
- 38. Coverage does not prove that you have TESTED every line. All it proves is that you have EXECUTED every line. And that is a big difference.
- 39. public class Calculator { public int sum(int a, int b) { return a + b; } }
- 41. Richard Lipton found the good answer in 1971
- 46. "If you want to know if a test suite has properly checked some code, introduce a bug"
- 47. return a + b;
- 48. return a - b;
- 49. Is it the whole idea behind the mutation testing?
- 50. Actually, Yes!
- 51. ● Artificial bug: MUTATION OPERATION
- 52. ● Artificial bug: MUTATION OPERATION ● Code with artificial bug: MUTANT
- 53. ● Artificial bug: MUTATION OPERATION ● Code with artificial bug: MUTANT ● When the test fails on mutant: KILLED the mutant
- 54. ● Artificial bug: MUTATION OPERATION ● Code with artificial bug: MUTANT ● When the test fails on mutant: KILLED the mutant ● When the test succeeds on mutant: mutant SURVIVED
- 55. return a - b; KILLED SURVIVED
- 56. killing is good
- 57. My code is much more complex than adding two numbers!
- 59. Math + to - * to / - to + / to *
- 60. Boundary < to <= > to >= <= to < <= to >
- 61. Negate == to != >= to <
- 63. Complex Remove IFs Remove method calls
- 64. Complex Remove IFs Modify return Remove method calls
- 65. Complex Remove IFs Modify constantsModify return Remove method calls
- 71. Do I need to change the code on my own?
- 73. public class Calculator { public int sum(int a, int b) { return a + b; } } @Test public void force100PercentCoverage() { calculator.sum(0, 0); }
- 74. DEMO
- 75. > MathMutator >> Generated 1 Killed 0 (0%) > KILLED 0 SURVIVED 1 > TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 > STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
- 76. @Test public void Should_ReturnSum_When_SummingTwoValues() { // given int a = 1; int b = 2; // when int result = calculator.sum(a, b); // then assertThat(result).isEqualTo(3); }
- 77. > MathMutator >> Generated 1 Killed 1 (100%) > KILLED 1 SURVIVED 0 > TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 > STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
- 78. Do I need to read the console?
- 79. DEMO
- 81. Hazelcast Kubernetes Plugin ● 5000 LOC ● 12 classes
- 82. DEMO
- 83. Why NASA engineers didn't use it in 1998?
- 88. if (a >= 100) { throw new IllegalArgumentException(); } if (a < 100) { fooBar(); }
- 89. if (a >= 100) { throw new IllegalArgumentException(); } if (a < 100) { fooBar(); }
- 93. return a + b;
- 94. return a + b;
- 95. return a + b; return a - b return 0
- 96. return a + b; return a - b return 0
- 97. :)
- 103. Main class paradox
- 106. // given
- 107. // given // when
- 108. // given // when ?
- 109. What to do with people who write tests without assertions?
- 113. Do people really use mutation testing?
- 116. Test your tests