Mateusz Bryła - Mutation testing
- 3. Woot? EXAMPLE 1 public void testMethod() { Something something = new Something(); something.executeSomeMethod(); } EXAMPLE 2 public void testConstructor() { Something something = new Something(); }
- 4. LITTLE HISTORY FIRST PROPOSED in 1971 by Richard Lipton A. Jefferson Offutt and R. H. Untch “Mutation 2000: Uniting the Orthogonal” FIRST DEFINED in 1978 R. A. De Millo, R. J. Lipton, F. G. Sayward “Hints on test data selection” FIRST IMPLEMENTED in 1980 by Timothy Budd T. A. Budd, “Mutation Analysis of Program Test Data”
- 5. PROBLEM
- 6. SOLUTION
- 7. EXPECTATIONS
- 8. TESTS - TDD?
- 9. TESTS - NO TDD?
- 10. TEST
- 11. MUTATION
- 13. EXPECTED RESULT
- 14. SOMETHING WRONG?
- 16. FLOW 1. Run tests 2. Mutate production code 3. Run tests again 4. Generate report
- 17. KILL THE MUTANT!
- 18. TESTS RESULTS? PASSED (SURVIVED) FAILED (KILLED) other results applicable (e. g. timeout, memory error)
- 19. EXAMPLE ORIGINAL if (i >= 0) { return "foo"; } else { return "bar"; } MUTATED if (i > 0) { return "foo"; } else { return "bar"; } conditionals boundary mutation
- 20. COMMON MUTATORS #1 CONDITIONALS MUTATORS < -> <= <= -> < > -> >= >= -> > == -> != != -> == MATH MUTATORS + -> - - -> + >> -> << << -> >> * -> / / -> *
- 21. COMMON MUTATORS #2 RETURN VALUES MUTATOR returns true if original returns false returns false if original returns true returns 0 if original returns int != 0 returns 1 if original returns 0 METHOD CALLS MUTATOR removes void method call removes non-void method call, returns default value for the return type replaces members with null in constructors
- 23. HOW TO GET STARTED? PITEST for JAVA http://pitest.org Parasoft INSURE++ for C/C++ https://www.parasoft.com/product/insure/ MutPy for Python https://pypi.python.org/pypi/MutPy/0.4.0 Maven: mvn org.pitest:pitest-maven:mutationCoverage
- 24. PERFORMANCE 100 classes 10 tests / class 1 ms / test tTOTAL = 100 x 10 x 1ms = 1s 10 mutations / class tALL = 10 x 100 x 1s ~ 17min tSMART = 10 x 100 x (10 x 1ms) = 10s 1000 classes 10 tests / class 1 ms / test tTOTAL = 1000 x 10 x 1ms = 10s 10 mutations / class tALL = 10 x 1000 x 10s ~ 28h tSMART = 10 x 1000 x (10 x 1ms) ~ 2min real life example: 481 test cases fully mutated on 16GB i7 -> 03:24 min
- 26. ATTRIBUTIONS Hands Up - Avel Chuklanov - https://unsplash.com/photos/9cx4-QowgLc (CC0 1.0) Bridge - Franc - https://thenounproject.com/search/?q=bridge&i=122542 (CC BY 3.0 US) Cargo Truck - No way - https://thenounproject.com/search/?q=truck&i=154864 (CC0 1.0) Philosoraptor - OnlyOneKenobi - http://wallpaperswide.com/philosoraptor-wallpapers.html Chainsaw - James Keuning - https://thenounproject.com/search/?q=chainsaw&i=11328 (CC0 1.0) Refugees - Gerald Wildmoser - https://thenounproject.com/search/?q=drown&i=208787 (CC BY 3.0 US) Resurrection - Isabel Martinez Isabel - https://thenounproject.com/search/?q=survived&i=202842 (CC BY 3.0 US) Pac Man - iconoci - https://thenounproject.com/search/?q=pac+man&i=9215 (CC BY 3.0 US) Stopwatch - Nick Holroyd - https://thenounproject.com/search/?q=stopwatch&i=12666 (CC BY 3.0 US) Zombie analysis - clement127 - https://www.flickr.com/photos/clement127/16089737804/ (CC BY-NC-ND 2.0) in order of appearance
Editor's Notes
- #2: Welcome everyone!
- #3: Show of hands! Who writes unit tests? Who believes they are worthy? Who measures coverage? Who has coverage at >= 85%? Who thinks coverage is the perfect metric? Nice tool. But it isn’t perfect.
- #5: 1971 - proposed by a student RICHARD LIPTON as a term work. 1978 - formal definition 1980 - first implementation But TOO SLOW!
- #19: PASSED - negative scenario FAILED - positive scenario
- #23: Story 1 - forgot to clear map - show example Story 2 - rare fault handling - one boolean method returning failure handling result - all unit tests expected false as return method - in mocking nightmare