Every code coverage tool is measuring the wrong thing (on purpose)
Download as PPTX, PDF1 like392 views
The document discusses the shortcomings of traditional code coverage metrics and proposes a quantitative measure based on the lines of code executed during tests. It emphasizes that while code coverage is valuable, relying on rigid coverage percentages can be misleading and that manual checks are preferable. Additionally, it addresses the relevance of mutation testing and the nuances of testing getters and setters.
Every code coverage tool is measuring the wrong thing (on purpose)
- 1. coverage tool is measuring the wrong thing (on purpose)
- 2. A deep(-ish) dive into code coverage
- 4. Who uses code coverage now?
- 5. Why use code coverage? “We want to know how well our code is tested”
- 6. Code coverage is good• Testing code coverage is good • How it’s done is often not so good • Why is that?
- 7. The Starting Point• “How well is our code tested?” • This is a qualitative measure • Computers don’t do qualitative • Can we make it quantitive?
- 8. A Quantitive Measure “How many lines* of code can I delete without causing any tests** to fail?” *statements, methods, branches, et **or compilation
- 9. Why is this a good measure? • Direct translation of the qualitative question • Makes sense • Minimises code written for a set of tests
- 10. This is expensive• Really, really, expensive • n statements/branches/method s = n(n-1) compile and test cycles • We need something cheaper
- 11. Downgrade from business class to cattle class
- 12. Find a lower cost approximation
- 13. A (low-budget) Quantitive Measure “How many lines* of code are executed when all of the tests are run?” *statements, methods, branches, et
- 14. A (low-budget) Quantitive Measure • Much cheaper • Approximately* the same thing * All the fun happens here!
- 15. The differences
- 16. Problem areas • Synthetic methods • Things you do to “make it compile” • Java 7 features • Useless code
- 17. Learnings • All code coverage engines are flawed • Some are profoundly flawed • It’s possible to lower coverage by deleting untested code
- 18. Proof
- 19. Proof
- 20. Learnings • You cannot reliably enforce: • “X% or higher coverage” • “Coverage always goes up”
- 21. Automatic enforcement of coverage levels is troublesome (Unless the coverage level is 1
- 22. What to do instead? • Spot check • Manually use the more stringent measure • Compare to last week, not last commit • If the number goes down, know why • Separate covered and uncovered
- 23. Two more things
- 24. Should I test getters and setters? • No • But… • Delete getters and setters that you can delete without making a test fail
- 25. Should I enforce 100% code coverage? • It depends… • Why you’re doing it • Who decides • If it feels like work
- 27. What is mutation testing? • Mutate statements instead of deleting them • Every mutation should make a test fail
- 28. Thoughts on mutation testing • Seems decent for loop logic or math logic • Doesn’t know how to mutate a lot of statements • Doesn’t mutate source code, just object code • Based on a traditional coverage run
- 29. UNITED KINGDOM +44 203 603 7830 helloUK@equalexperts.com Equal Experts UK Ltd 30 Brock Street London NW1 3FG INDIA +91 20 6607 7763 helloIndia@equalexperts.com Equal Experts India Private Ltd Office No. 4-C Cerebrum IT Park No. B3 Kumar City, Kalyani Nagar Pune, 411006 CANADA +1 403 775 4861 helloCanada@equalexperts.com Equal Experts Devices Inc 205 - 279 Midpark way S.E. T2X 1M2 Calgary, Alberta PORTUGAL +351 211 378 414 helloPortugal@equalexperts.com Equal Experts Portugal Avenida Dom João II, Nº35 Edificio Infante 11ºA 1990-083 Parque das Nações Lisboa – Portugal USA helloUSA@equalexperts.com Equal Experts Inc 315 Hudson Street 9th Floor New York City, NY 10013 Thanks!
Editor's Notes
- #5: And why? Sometimes, this is “because my boss makes me”, but setting that aside…
- #7: This is a thing I hope most of us can get behind. The problems are not that this is done, but how it’s done. Sometimes this are people problems (we’ll get into that later), but sometimes they aren’t.
- #9: This isn’t the only measure… “can I change a statement without making a test fail?” is also a good one?
- #11: The sample java later in this presentation has 23 statements that could potentially be removed (excluding trivial things like throws clauses and import statements). That’s 506 build and test cycles.
- #15: 1 instrumented test run, which is more expensive than normal, but cheaper than hundreds or thousands of test runs
- #17: Synthetic methods: Default constructors, methods on enums Java 7: ARM blocks. The compiler purposefully puts in more blocks than will be executed (null checks in the finally, etc) knowing that the JIT will optimise the extra ones away. Coverage tools don’t even attempt to detect useless code.
- #18: Profoundly flawed = Java 7 support, etc. Delete an untested method that does nothing but was executed during a test… coverage goes down slightly. In our example, with the spurious method, instruction coverage is 82%. Without it, coverage is 70%.
- #20: Note that branch coverage went from 100% to undefined!
- #21: Profoundly flawed = Java 7 support, etc.
- #22: If it’s 100%, and you delete a chunk of untested code, it should still be 100%… because all of the code that’s less should still be covered. This also holds for 0% coverage. I assume we’re all happy to ignore that case.
- #23: Separate code: consider a module with 100% (or high) coverage, and another module without enforced coverage. Move things into the one module over time.
- #25: Trivial getters and setters don’t need to be tested directly. Tests are executable documentation, and documentation isn’t needed for that.
- #26: should you enforce 100% code coverage? twice in my career I’ve been on teams where we were close to 100% code coverage. In 2013 we were three or four statements/branches away for a while. Spot checking every week or so. So finally, we put in explicit tests to cover those three or four spots. A week later, we were still at 100%. I talked to some of the guys on the team… should we fail the build if coverage isn’t 100%? Let’s try it.. see what happens. We turned it on, and forgot about it for a couple of weeks. Then the first few times we tripped it, it was definitely areas where we had forgotten to write a test… so we decided to keep it. Also, when somebody asked “what’s your code coverage?” and you can say 100% without checking anything you feel like an absolute boss. Good for political reasons sometimes. :-)
- #27: should you enforce 100% code coverage? twice in my career I’ve been on teams where we were close to 100% code coverage. In 2013 we were three or four statements/branches away for a while. Spot checking every weak-ish. So finally, we put in explicit tests to cover those three or four spots. A week later, we were still at 100%. I talked to some of the guys on the team… should we fail the build if coverage isn’t 100%? Let’s try it.. see what happens. We turned it on, and forgot about it for a couple of weeks. Then the first few times we tripped it, it was definitely areas where we had forgotten to write a test… so we decided to keep it. Also, when somebody asked “what’s your code coverage?” and you can say 100% without checking anything you feel like an absolute boss. Good for political reasons sometimes. :-)
- #28: The last two times I’ve done this talk, people have mentioned mutation testing — specifically PIT. (Which seems to be the viable option in the Java world)
- #29: Example mutations: return null instead of a value, subtract instead of add, that sort of thing
- #30: The class of problem PIT is really good at catching is tests that don’t assert anything. To improve performance, PIT does a single traditional coverage run… which it then uses to learn which tests to run which mutations against. Which means it’s got a gap for statements that aren’t executed by any tests…. Same old problem. Mutating object code and not source code means that we can’t see that a mutation doesn’t make something not compile. False positives mean that improving code can still make the coverage percentage go down. An example of this would be removing one of two duplicate methods.