SubTests are the Best

Editor's Notes

• #3: Me Developing in python for a few years Previously I’ve worked as a math teacher Caktus Web development company in Durham, North Carolina Specialize in web apps, usually built in Django Other python projects
• #4: Tests are there to make sure the code that we’re testing actually works Upgrading to new versions of libraries is much easier when we have tests - that way we know when upgrades break things, or even when things are going to be deprecated in the future If I can catch a bug in my code by writing unit tests, then somebody else's time isn't used up in finding these bugs during QA. It may seem like it’s faster to just not write tests, but forcing myself to write unit tests has allowed me to catch many bugs in my code. This way I can fix those bugs now instead of waiting for QA, then fixing it later. It's better to write a test and catch a bug, than to have a customer catch a bug in production
• #5: but this talk isn't about why we should write tests. It's about subtests. First, a little more about testing
• #6: readable: other developers understand them and can update them when updates are needed thorough: test multiple things about a function/class/endpoint DRY: We shouldn't be writing the same lines code in multiple places By the way these principles taken from the zen of python: “Readability counts.”, “Beautiful is better than ugly.”, “Explicit is better than implicit.”, and “There should be one—and preferably only one—obvious way to do it.” The rest of this talk is about making tests readable, thorough, and DRY, and specifically how subtests allow us to do these things. For people who have not seen subtests before, they are a way to break up tests into smaller sections within a single test. Why would you want to do that? Because it makes your tests more readable, allows you to be thorough, and keeps your tests DRY.
• #8: Let's say we have a site with users, they each have a profile, and they can follow each other. In order to make sure we're tracking following correctly, we should write a test. We might begin with something like this. We create 3 profiles, and verify that when a profile is created, no one is following anyone else. But to add testing for when people start following others, we can add some more code
• #9: So we added a section that has profile1 following profile2 and profile3 following profile1. We should also include a section on unfollowing, so let's add that As a note, I’m not explaining each line, just a high level of what we’re doing to create the test.
• #10: So now we're asserting that no one is following anyone at the beginning, that adding a follower tracks followers correctly, and removing a follower tracks followers correctly. But now the test is pretty long, and if someone else wanted to read it or edit it, it would take some time to figure out everything that is being asserted. Since it really has 3 sections, let’s split it up into 3 sections
• #11: Now we can see that there’s a setup section, a section with asserts, a couple thing change and second section with asserts, and something changes and a third section of asserts. We can make it better with comments for each section.
• #12: The text is small, but now each section has a comment at the beginning. However, there are no comments about the database changes or any of the asserts. A better example:
• #13: Now a quick look at the test shows that it has some setup at the top, and then 3 sections, each with comments about what is happening. Again, it’s not important to be able to see everything that the test is doing, but a quick look at the test shows that is has 3 sections, and the subtest comments explain what is happening in each section.
• #14: Readability important: Easier to find what’s happening (especially if I come back to this code in 6 months, or if someone else needs to update the tests) More pleasant to read (people, like me, like to figure out what’s happening in the code quickly, instead of digging around for a long time). If the code is easy to read, then I can spend my time updating it, rather than trying to figure out what it’s doing. For companies, these lead to better code, better projects, and more efficiency….more money.
• #15: That’s not a bad idea, and it will work to do that. For example, we could have taken each of those 3 sections and made them into their own test. However, the downside to this approach is that we have to do the same setup for each of those tests, and even more for the one where we are testing the removal of followers. Sometimes it’s not a big deal to break up one test method into many small test methods, but other times the setup we have to do is redundant, and not DRY. For instance if we have a test that requires the creation of objects from 5 different models in our test database, it would be a bad idea to repeat the same setup steps for each test. Instead, we can do the setup once, and use subtests.
• #17: Let’s say we have a function, is_user_error(status_code) that returns True or False when given a status code. Should be True for any 400-level status code. One way to write tests for this function would be to write some of the most common codes into the tests. This test works and it tests a lot of the common cases. It even gives our function 100% code coverage. But this is incomplete. There are many other status codes that we’re not testing
• #18: So, should we put in a line for each status code that might come up? That would be thorough...but not feasable.
• #19: Instead, we can write a loop and go through each status code, but if one fails...
• #20: We get a non-descript message. Some code failed, but we don’t know which one. Using a subtest parameter, we can do:
• #21: We’re passing in the status code that we’re testing as a parameter into each subtest, so if something fails, we get:
• #22: And we see clearly that the error is for status code 405. Ok, this was a simple example, and probably not something we would need to test in a real app, but using parameters allows us to track the value of the parameter in the subtest, and as a result we have both thorough tests and specific error messages. And we can pass in any number of parameters, so when something fails, each of those parameters gets output back for us to see. Another thing to note is that subtests run independently, so if status codes 403 and 405 both failed, we would get a failure for each, whereas without using subtests we would only get the first failure.
• #23: This can actually be really helpful in diagnosing the problem in an application. If we see all of the failures at once and can see something similar for them, we may be able to diagnose the bug quickly, instead of having to work with only the information for the first failure. Here’s an example:
• #25: Based on the parameters we can see that something is not right for when a person has an empty string for a first or last name, and we can quickly get on to fixing the bug. By the way, could we have passed in an assert message with those parameters? Sure, but subtests don’t have to rely on custom assert messages. We can just pass in the parameters, and they show up when something fails, and we can still use a custom assert message for anything else we may need it for. Not having to use custom assert messages in each test means that we save time, the code is DRYer, and easier to maintain, since we don’t have to read through each message when working on tests.
• #26: It’s important to test all the parts of a function, not just one part of it, or that it exists. We want to actually verify that our code works, so we need to be thorough. Subtests allow us to be thorough, as well as readable, and give us more useful error messages because they are independent and can print out the parameters when they fail.
• #28: Let’s say we’re testing an API endpoint, and we want to make sure that the right fields are required. There is first_name, last_name, and address. So, we write a test that POSTs to the endpoint with each of the fields missing, and we verify the response status code is that of an error. So we can ask a few questions about this test: Is it clear what’s being tested? Is the test DRY? Can we improve it? The answer is, yes, we can improve it
• #29: Here’s the same test with some line breaks, and some comments.This is not bad, and with the comments on top of each section it’s relatively clear what’s happening. But really, we’re repeating the same thing 3 times, which is not DRY. Instead, we should have define this behavior in 1 place, and each subtest can test each missing field name:
• #30: Now we define a helper method, get_minimum_required_data() that returns the data that is required for a successful POST. This is defined in 1 place, so it’s DRY. The test for missing data now loops through each case of missing fields, and tries to POST without that field, then asserts that the status code is 400. Why is this better? DRYer code If something about the required fields for the API endpoint changes, we only need to change it in 1 place in the tests Each case is defined neatly in one tuple. If we need to change it, we can do it quickly. Since things are defined neatly, it’s more readable. If we need to add more fields, that’s easy to do as well For another API endpoint we might have 10 fields that are required. It’s much better to just add them in one place, instead of writing the same test 10 times.
• #31: Less code to read by me and other developers It’s easier to maintain, since we define things in 1 place, rather than many Subtests give us a great way to test in a way that is DRY
• #32: Readablity matters, since someone will need to read through our tests in the future subtests make it easier to have readable tests, since they provide a way to break up tests into sections, and to have comments for each section It’s important to have thorough tests. We don’t want to just test a little bit of the functionality, but a reasonable amount. Subtests allow us to have thorough tests without them being really long DRY tests are important Subtests give us another tool for having DRY testing At Caktus we strive to have good code, testing, and documentation, and subtests have been helpful in allowing us to do this.
• #33: No.
• #34: You can still write tests with no comments or bad comments, with confusing variable names, with lots and lots of unneccesary setup, etc. However, subTests give us ways to avoid some of these things, and we can use subTests as one of a number of tools to write readable, thorough, and DRY code.
• #35: Can you do all of this without subtests? Pretty much. There are multiple libraries like py.test that allow us to write readable, thorough, and DRY tests. However, there’s a way to do this in the standard python library without relying on any specific testing library. If you want to use those libraries, go for it! But subtests are always there as a tool that you can use in your testing. Whether you use them or not, they are available to help you write readable, thorough, and DRY tests. Thank you.