What You are Doing Wrong with Automated Testing
- 1. What You are Doing Wrong With Automated Testing Anti-Patterns in Automated Testing by Martin Folkoff and Shawn Faunce
- 2. US Martin Folkoff US CIS Shawn Faunce
- 3. Anti-Patterns • “An anti-pattern is just like a pattern, except that instead of a solution, it gives something that looks superficially like a solution but isn't one.” - Andrew Koenig Ice Cream Cone Recorders Low Priority Bugs OwnershipMonolith Coverage = Quality
- 4. Ice Cream Cone 99% UI Tests 70% Unit & Integration Tests 20% API Tests 10% UI Tests Exploratory Testing Looks like a good solution • Easy to create • Mimics user interactions • You see something happening 10% Unit & Integration Tests 20% API Tests 70% UI Tests Exploratory Testing Anti-Pattern: When you automated testing strategy relies mostly or exclusively tests at the user interface (UI) layer But it introduces problems • Maintenance nightmare • No fast feedback • Complex tests, complex debugging
- 5. Looks like a good solution • Tools make it quick and easy • No programming required • Leverage manual testers Anti-Pattern: When you use a screen recorder to build your automated tests instead of one of the automated testing frameworks that relies on programming. But it introduces problems • Extremely fragile • Dependent on having a UI • No mock objects Recorders
- 6. Looks like a good solution • There are bigger fish to fry • Only happens occasionally • Not worth our attention Anti-Pattern: When you ignore intermittent low priority bugs that occasionally affects the automated test, but a user might easily overcome or not notice. But it introduces problems • Lose confidence in your tests • Flaky tests • Build technical debt Low Priority Bugs
- 7. Looks like a good solution • Avoid managing dependencies • Always run your full test suite • Just keep appending new tests Anti-Pattern: When you organize your test suite so that all of the tests are designed to run together as one test suite. But it introduces problems • Slow feedback loop • Longer build times • Encourages sloppy engineering Monolith
- 8. Looks like a good solution • Testers test, and this is a test • Frees up developers • Clear lines of ownership Anti-Pattern: When you have testers write automated tests, you are transferring the ownership of quality But it introduces problems • Need programming skills • Quality is not someone else’s job • Creates ownership issues Ownership
- 9. Looks like a good solution • Can’t get any better than 100% • Everything appears to be tested • The dashboard is all green Anti-Pattern: You achieve 100% test coverage on your code, ergo you have 100% Quality. But it introduces problems • Are the tests good tests? • Satisfy the coverage tool • Can lead to assertion free testing Coverage = Quality “I expect a high level of coverage. Sometimes managers require one. There’s a subtle difference.” “People optimize their performance according to how they’re measured” “If a part of your test suite is weak in a way that coverage can detect, it’s likely also weak in a way coverage can’t detect.” -- Brian Marick How to Misuse Code Coverage http://www.exampler.com/testing-com/writings/coverage.pdf