How good are your tests?
0 likes269 views
Mutation testing is a technique to evaluate test quality by automatically creating small variations of the production code, called "mutants", and then running the test cases against these mutants to see if the tests can identify the right outcome. It is more effective than just looking at code coverage alone. Mutation testing with PIT is easy to use and configurable, allowing different types of mutations to be introduced and test runs to be optimized for speed.
How good are your tests?
- 1. How good are your tests? Noam Shaish
- 2. How do we test? • Unit testing • Continues Integration • Test-drive development (TDD) • Code coverage
- 3. Unit testing • Prove code works • Instant regression tests • Improve code design
- 4. Continous Integration • Automated testing • Maintain single source repository • Collect statistics
- 5. All tests passed So f*cking what?
- 6. Test that always pass
- 7. Code coverage • Measures line/branches executed during testing • Common way to estimate testing
- 8. We are all covered • Are we?
- 9. Test that gives full coverage
- 10. So how good are your tests?
- 11. Mutation testing • Proposed by Richard J. Lipton in 1971 • A way to measure the quality of your tests
- 12. Mutation • A small change in your code
- 13. Mutant • A mutated version of your class
- 14. Mutation testing • Create mutants • Run tests • Check outcomes
- 15. Outcomes: killed • Mutant killed: Mutant is killed if a test fails, which proves the mutated code is properly tested
- 16. Outcomes: lives • Mutant lives: Mutant didn't fail any test…
- 17. Outcomes: timed out • Mutant timed out: Mutant caused program to loop or stuck
- 18. Outcomes: other • There there possible outcome but less relevant
- 19. Mutation testing with PIT • http://pitest.org • Configurable ‘Mutators’ • ASM (byte-code manipulation) is used for manipulation • No mutation storage • Reports
- 20. Mutators: Condition Boundary > into >= < into <= >= into > <= into <
- 21. Mutators: Negate Conditions == into != != into == >= into < <= into > < into >= > into <=
- 22. Mutators: Remove Condition if ( a == b) { attack(); } into if ( true ) { attack(); }
- 23. Mutators: Math + into - - into + * into / / into * % into * & into | >> into << << into >> >>> into <<< a++ into a— a— into a++
- 24. Mutators: Others • Replace return values (return a; into return 0;) • Remove void calls (attack(); is removed) • Replace value by null • Some mutators enabled by default and some aren't
- 25. Mutation testing speed • Mutation testing is CPU intensive • Multiple test runs • PIT has a lot of speed optimizations
- 26. Which test to run? • PIT uses code coverage to decide which test to run • A mutation is created only on lines covered by 3 test
- 27. Summary • Mutation testing automatically tests your tests • Mutation testing can find bugs in your tests • Code coverage is half the story, and gives a false sense of security • Mutation testing with PIT is easy
- 28. Show me some code