Level Up Your Automated Tests
Download as PPTX, PDF1 like1,662 views
The document discusses ways to improve automated testing practices, such as by making tests easier to write, more readable, and more meaningful. It suggests using mocking frameworks to reduce test complexity, following standards and examples, and adopting a testing framework like Spock that addresses common issues. While progress has been made, there is still work to be done to encourage writing tests, make tests easy to write and understand, and ensure tests are meaningful and focused on the purposes of testing.
Level Up Your Automated Tests
- 1. Trisha Gee (@trisha_gee) Developer & Technical Advocate, JetBrains Level Up Your Automated Tests
- 2. Using <Technology X> Will Fix Your Problems
- 3. •Write Tests? •Write Readable Tests? •Write Meaningful Tests? How can we:
- 4. Why Don’t We Write Tests?
- 6. What Are Tests For?
- 8. What Are Tests Actually Good For?
- 10. How Can We Change Attitudes?
- 11. How Can We Change Behaviour?
- 13. Quality becomes a habit
- 14. This can only get you so far
- 26. It needs to be easy
- 27. Possible Solutions •EasyMock / Mockito / JMock •Home-grown mocking / stubbing •Hamcrest matchers •DSL – Domain Specific Language •Standards / Examples
- 28. Along came Spock
- 29. An Example Test
- 30. How it fixes our problems
- 31. Hard to read
- 34. Lots of setup
- 35. No Unit Tests
- 37. Too many similar tests
- 38. Testing too many things
- 39. Testing too many things
- 41. Proving itself
- 43. Remaining Issues
- 48. Still more to be done
- 49. a) write tests? How can we:
- 50. b) write readable tests? How can we:
- 51. c) write meaningful tests? How can we:
- 52. Conclusions
- 53. Make it easy
- 55. Zero tolerance for failures
- 56. Have a champion
- 57. Let it go
- 58. Pairing or code review
- 59. Focus on the purposes of testing
Editor's Notes
- #2: This is a story of how to get more out of your automated tests This is my experiences at MongoDB, not JetBrains We had tests, we had coverage, but it felt more like a box-ticking exercise than a commitment to the value of tests or testing. And then we started using Spock, and magically everything got better
- #3: This is NOT one of those talks Instead, I’m going to talk about the problems we were facing And some of the solutions that helped us, yes including some technology choices But we’ll see WHY they helped us
- #5: We know testing is important, yet we still don’t do it when we should
- #6: Common reasons why not 1 Too hard to setup 2 Single use code 3 The code is simple 4 Too hard to mock 5 UI code 6 Not enough time 7 Effort vs Value 8 I'm the only maintainer
- #7: Let’s take a step back and look at a different question
- #13: Me Asking why Jenkins was failing Fixing tests Adding checkstyle Fixing up checkstyle errors Dishing out remaining errors Pushing for tests in code reviews
- #14: Get things green Require tests for code review Got use to making sure Jenkins and travis were green Got used to writing tests Reports are useful too
- #15: There comes a point where it’s too hard to get the team to do everything you want You want it to be easy and logical for others to know what to do and how to do it The champion’s job is to put into place processes, practices, examples that become second nature to the team - checkstyle a success - unit testing not so much I kept wanting to re-write other’s tests, to add more tests. Good that our code review process didn’t allow that, or I would have become a tester
- #16: Champion and good habits are a start Helped us address writing tests and writing readable tests to some extent We still had problems Some technical, some around improving quality (meaningfulness) of tests)
- #17: Insert Jenkins matrix?
- #26: MongoClientURITest
- #27: Most of all It needs to be automatic for all the developers
- #29: Found out about it at a conference like this one Was sceptical a library could solve what I thought were behavioural problems Addresses quite a lot of the issues we had Week of Spike
- #30: // String method names // given/when/then // Map syntax // Type coercing // new BasicDBObject("$set", new BasicDBObject("x", 2) // equals
- #31: Halfway
- #34: Two things - @Subject - “when”
- #35: Setup is explicit When the setup section gets big, sign that we should refactor (not just a Spock solution) Use “Fixture” for example – testDollarOutOnSecondary
- #39: Same problem, different side of the coin
- #42: - Catching server bugs - Catching regressions - Checking combinations of servers & drivers & java - How the team embraced it – using Spock for integration and API-level tests - Jenkins and Travis are now generally green, test coverage is up, tests written for new sections (async) Not driven by me: - Introduction of server specs - Then test specs to prove a driver meets the criteria - In future, would be great to have a TCK-like set of tests that can prove a driver meets MongoDB requirements (or some set of them)
- #43: - Catching server bugs - Catching regressions - Checking combinations of servers & drivers & java - How the team embraced it – using Spock for integration and API-level tests - Jenkins and Travis are now generally green, test coverage is up, tests written for new sections (async) Not driven by me: - Introduction of server specs - Then test specs to prove a driver meets the criteria - In future, would be great to have a TCK-like set of tests that can prove a driver meets MongoDB requirements (or some set of them)
- #44: Other problems - Performance - Dynamic lang has downsides - Now have even more inconsistency in our tests (but at least we know which are the new ones) - Refactoring – not so awesome with Spock - Still not clear to people what the different tests are for – unit, functional, integration, acceptance
- #45: Perf is worse for running individual tests (Groovy has a startup cost) Perf is slower to run the overall build (Groovy, plus adding an extra gradle step) Perf is not RELIABLE – performance critical tests (i.e. with timeouts) needed to be Java
- #46: - Masks errors - Finding problems at runtime instead of compile-time - (On the other hand, this is really useful when you don’t care about types or nulls or whatever) - Refactoring – not so awesome with Spock
- #47: Now we have old tests, good junit, spock (unit), spock (functional)
- #48: Expected spock to be used for unit only Then we could be clear about spock – single class or a couple of classes Now we’re not really sure about the difference between unit, functional, acceptance.
- #49: What I’d like it to do - I still want tests that document the system - I’d like automated perf tests, but production-like data is too hard - I’d like soak tests - I’d like much more complicated tests, which do more than one thing (is it not enough to test each piece individually?) - I’d like more unit tests - Basically want more types of tests, with better clarity on the purpose of each test type. Originally hoped Spock would be for unit, and all integration tests would be truly end-to-end and Java
- #50: - Pairing - Code Reviews - Make it simple - Make it fun - Understand WHY we test - *Could* use coverage
- #51: Groovy helped us - DSLs - Gwen - even (shock) comments - good test names - test one thing (have single reason for failure?) - make it clear what’s under test, what’s expectations - extract common functions (connect to DB, get server, etc) - Have good example tests for people to follow (guidelines) - Make tests a first class citizen (code review for example) - This is all just common code practices, it’s the same for tests as it is for production code. Somehow, we fail to do this in tests
- #52: - This might actually be the hardest thing to do. Define meaning - Code review - Understand the purpose of the test – end-to-end? Unit? Checking the server? Checking your assumptions? Documentation? Requirements? - Tester mindset – What if? What does this mean? What are the implications? - |Good idea to get someone who isn’t close to the code or the system to check out the tes
- #53: Suspect coverage is the wrong metric to drive things A new language might be a fun way to get developers to pay attention to tests (or it might be a hindrance)
- #55: Jenkins Static analysis Coverage
- #56: Either fix it Or delete it
- #57: Have a champion – someone needs to give a shit And they need to not get annoyed at pushing this all the time
- #58: Don’t be religious Don’t hold tight to ideals, go with what works (Spock not just for unit; having to let go of acceptance tests)
- #59: Or code showcase