Test Design for Continuous Integration and Delivery (2020)

Editor's Notes

• #3: Talk about Microsoft and Bing especiallyKey things to hit:* Worked with continuous builds in some form for my entire career Worked as an SDET, as a straight dev, and as a combined engineer Created this from scratch at PlayFab Talk about Microsoft and Bing especiallyKey things to hit:* Worked with continuous builds in some form for my entire career Worked as an SDET, as a straight dev, and as a combined engineer Created this from scratch at PlayFab
• #4: This is an automated pipeline! Don’t get too in-depth, we will talk moreTalk notes:* Please speak up if there’s a question Feel free to ask questions at any time, or for examples or to deep-dive* Ask about experience in the room – who is working greenfield? Updating an old existing code base? Working in an existing CI/CD pipeline? What we are going to talk about: What makes a good test case, when we want to run those test cases, and then how we monitor and report on those test cases
• #6: These are my guiding principles of test design – ESPECIALLY KEY when things are fully-automated
• #8: * Don’t bog down on what these mean, but give a high-level overview
• #9: In a perfect world, we would always run every test In a perfect world, every test would be super fast and we would also have infinite time Battlefield triage is key!
• #10: If you had time to only debug one of these two issues, which would you do? The ‘time game’ is a fun way to figure out what tests really matter
• #11: Squeaky tests only train people to ignore test failures – and then they WILL ignore a real failure and it will go to prod by accident – ask me how I know! Try – finally blocks are amazing – no catches required! Any test that touches the file system or DB system must do that. Obviously – a test can’t be automated if it requires someone to manually purge the DB after it runs
• #12: Sometimes tools can help you find flaky tests
• #13: Be ruthless with cutting out overlapping coverage – read your code coverage reports and slash away! Can be nerve-wracking to delete tests Very tempting to roll lots of things into a single test case to save time, but this makes debugging a nightmare
• #15: * Watch out for global static variables. Watch out even more for a shared test database NEVER (almost never) write a test that relies on Thread.Sleep. Timing issues are horrible to debug! Can you make it synchronous? A callback?? Caching and shared resources can introduce the weirdest and hardest to debug test failures. Time-killers! Spend time to work around them beforehand instead of spending time debugging the failure (and then having to work around them anyways)* Talk about the angst from the shared test Dynamo at Playfab and the days spent figuring out what happened and how to stop it
• #16: The goal is to come up with lots and lots of test cases, which we will later be organizing into test suites DO NOT get into putting them into suites DO NOT throw away tests
• #18: This is an automated pipeline! Don’t get too in-depth, we will talk moreTalk notes:* Please speak up if there’s a question Feel free to ask questions at any time, or for examples or to deep-dive* Ask about experience in the room – who is working greenfield? Updating an old existing code base? Working in an existing CI/CD pipeline? What we are going to talk about: What makes a good test case, when we want to run those test cases, and then how we monitor and report on those test cases
• #19: Unit tests may cross functional boundaries, in that they may test more than one ‘function’. But they shouldn’t test an app.
• #20: * Simplest structure means running one build, one set of tests, and calling it done. Few projects can be that simple! Tests grow with scale! Breaking up a code base is half art and half science. (those are the bullet points) Ideally every test would run all the time, but almost nobody has time for that. Test what you can, and know what risk you are swallowing for later Make sure each stage fails quickly if it’s going to fail. Do NOT propagate failures down the pipe.
• #21: This is the sample architecture of a fictional service – say, something like that image hosting site from our last example. We’ll use this architecture as a guide for the talking about what tests to write and when to run them.
• #22: Many more potential test suites in these categories.
• #23: Remember – if you’re going to have manual steps, it’s in step 4.
• #24: This is a sequence diagram that describes the lifecycle of a single change.The tester is actually in control here – we are conditionally running different tests suites depending on their judgement of the change
• #25: ‘Integration’ test here is category. A number of test suites can fall into this category – your smoke test suite, for example.
• #28: Speciality is anything that can’t be run automatically and quickly. Maybe it requires manual setup, or maybe it’s difficult to have a machine to validate. Maybe it just takes a MILLION years to run. Maybe it is unreliable.
• #29: * let’s use all the tests that we came up with before, and now let’s prioritize and move them into test suites. * Let’s give each test suite a maximum wait time before a failure is cheaper than waiting for it to pass.
• #32: This graph is not scientific! In my experience, the sweet spot is somewhere between 70 and 80% code coverage. Any more than that is a whole lot of time for not much return. Line coverage just means a line was run,. Branch coverage includes proper handling of ifs and switches. It’s generally more diagnostic. Lets you know what needs to be tested more – a 5% covered package versus a 50% covered package Quanitify the output of your test efforts over time to management! They eat this stuff up (HA HA HA)
• #35: Which of these would you rather debug?