Making testability our mission

Editor's Notes

• #3: Im not sure testing is really working out that well. We always seem to be in the middle of existential angst. Agile, DevOps, what shall we fail to embrace next? Lets change our footing.
• #4: Lets add models here. Quadrants, THSM, Pyramid etc. Then a big cross. If its hard to test. Your strategy is for shit. Sorry.
• #5: Some people say this, but it is slightly less vague than quality is everyone’s responsibility. Describes what a bit better and who might be doing it. Still hard to aim at while building hard to test systems.
• #6: There is no bottleneck of the amount of testing, the real bottleneck is poor testability. Or to be more specific, the ability of the team to test and the testability of the system in question.
• #7: Ah ha! Now we are getting to it. We’ve heard this one right? Its a config change. A no-op release to pick up a new setting. Don’t you worry about it. ‘Just’ is one of my favourite words in testing. This actually means with all the forms of testing we currently do and I know about, we can’t imagine how to test this. Ability to test and testability meet head on. Guess what happens next.
• #8: I always enjoy this one. There is ALWAYS enough time to do any amount of testing. However, a better phrasing is the that in the time we have, we need to do the most effective testing for the risk at hand. Testability gives us this.
• #9: We’ve all worked on hard to test systems. If you think you haven’t then you are or were in such denial that you were a captive reliant on their captor. Stockholm syndrome. So, lets drop some truth bombs on our candy asses.
• #10: Simply won’t get tested - here’s a secret for you. IF IT HASN’T BEEN TESTED IT DOESN’T WORK, new stuff rarely does. You the tester will be burdened with it. Volunteers will be hard to find. Automation will target the areas that don’t break, or will just cover new stuff, rather than old fragile areas. Even if you can do performance and load testing, it will be brittle, late and on inappropriate environments. Probably mislead you more than lead you. One of the key indicators of poor testability is lack of diversity within your testing. IMPORTANT - your poor ops people. Sys Admin, DBA’s and App Support will be driven mad by your application. Hard to test means hard to operate in live, where it matters. Finally - your team will be divided by this system that you all hate, right down the lines of role. Unchecked, this will persist FOREVER.
• #11: Reverse inspiration Core concepts Testability Engineering Its about YOU
• #12: 2000 bugs in 2 years Communicated through tickets Longs test cycles against builds on long lived environments Mastered weirdly named tooling “Quality Centre” Left with a weird feeling - we did tons of testing, but we never got any faster, no one got what they wanted…
• #13: I thought testing was raising bugs, communicating via tickets
• #14: Superset as in, other ililities are contained within it Which makes it ethereal at times which is part of its problem, it his hard to describe but makes the world better. For me this is true of a lot of aspects of testing, where we co-opt other technologies to enhance our testing. One of the things that makes testability so intuitive as a direction for the craft of testing. But also makes it important, it is telling us to focus on the whole system, rather than making local optimisations. Let’s make this a bit more real, by talking about 4 core ilities of testability. In no particular order though.
• #15: Observability allows us to understand the system as it actually is - we can explore and ask questions of the system Observability determines what problems we can detect and how we evaluate if they are problems observability tools and techniques are the lens to view and filter that information. Tracing through a micro service architecture is a great example of this. Seeing the whole transaction throughout a set of dependent services. Great for seeing effects and side effects of a behaviour.
• #16: Controllability determines the dept and breath of our testing efforts - how deep you can go while still knowing what breadth you have covered. Without this you can go down the rabbit hole and miss the bigger picture. Without control testing is pushed later and later, Controllability determines what scenarios we can exercise - whether it be setting test data to the right state or ensuring a dependency returns a specific response.
• #17: The ability to isolate components from one another - to truly know the effects that either being connected to (or not connected to will have an impact on the component you are testing) - more importantly knowing that you can develop and test wherever you want and isolate problematic areas. Being able to isolate problems easily speeds the development effort, moving from guessing where problems are, to isolating components and interactions, to chase problems to their origin https://martinfowler.com/bliki/CircuitBreaker.html
• #18: The more complex the system, the harder it is to test. Sounds intuitive right? The harder it is to reason around a system, how many technology types, transport mechanisms, input, outputs, dependencies it has, the more problems can occur. Lots and lots of problems means lots of time spent testing, clarifying, checking, asking, exploring, re-exploring. You get the picture.
• #19: Lets address the question of how do we make testing more meaningful for all team members, as usual we need some principles to work by, without those we are attempting to change the world from no platform. These should also be inclusive of roles, whether your focus is humans, technology or a mixture of both.
• #20: A silent system is of limited use to anyone. We should be aware that for someone, somewhere, on some device your system is a pile of unusable rubbish that meets none of their needs. You just sleep well as you don’t know about it. Testability enables diverse testing activities, tests of differing speeds, testing when it matters most and giving actionable information. Lim ited forms of testing give you limit feedback. Risk accumulates with slow feedback right?
• #21: Everyone needs to collaborate more. But that needs a basis, especially across roles and disciplines. Testabaility is a joint endevaour. If you have been forged in a culture where devs do dev and testers do testing. Breaking down hen it isn’t too surprising when that doesn’t happen naturally. How can testability assist with that basis? Imagine you have a problematic dependency and two teams argue over it. Your service is always down! No, your integration is poor! This conversation enriched with feedback( from both systems is a much better one. The same goes within team. Better testability == better feedback == better relationships between roles. We are providing critique on work, even done well it is a balance we need to strike.
• #22: This is big. I’m going to make a statement here: Testable architectures are a leading indicator of quality. If an architecture is overly complex, has components that your team doesn’t control, you are heading for a fall. The flow of work through your team is dependent on architecture too. Supposed bottlenecks in testing are often bottleneck in architecture. Have a single team of DBA’s/Sys Admins responsible for your database/hosts? You will have a bottleneck that manifests in testing. Most systems have grown up over time, therefore serious diagnosis of the smells of hard to test architecture is needed. They don’t describe themselves as it.
• #23: Testability means that you are testing what matters most, when it matters most. The key here is who is involved in that statement can really help with the what and the when. In order to truly engage with the risks and rewards feared and valued by organisations, relationships between stakeholders are needed Sounds obvious right? Often the world isn’t shaped like this though. It wouldn’t be a talk about testing without something about the dark art of communication. To improve a system, you must learn its interactions, organizational relationships assist the flow of information between people and teams.
• #24: We must inherently accept that we cannot find all the problems right? Observability gives us this power to give our selves the best chance if finding the problems that matter. At a previous organization, we implemented New Relic, including its mobile front end agent to allow us to see client side errors. The avalanche of javascript errors was something to behold. Note the use of the word “helps” observability tools can see for us, but not interpret. The most common javascript error was mysterious. A stack trace is only a stack trace. The error was actually caused by losing network signal, which caused a much deeper error but the manifested error did not show that. An explorer was needed. But combined with the tooling, the real cause was found.
• #25: Have you ever got the feeling you aren’t really in control of the system under test? You exercise it in one way and strange things happen elsewhere. When you talk about your testing without being in control of your state you are providing bad information. We understand that when your system state is not controllable the side effects of testing cannot be observed, which compromise the effectiveness of your testing. For testing to give the most relevant and timely information, controllability is required. Same tooling and mechanisms where sensible for test environments and production.
• #26: If your system is hard to operate then it stands to reason that it will be hard to test. To be honest, its often where the problems really lie. As opposed to testing features over and over. I often hear this when asked to perform performance and load testing. Often systems are not operable enough to be tested in this way. They do not have logging, monitoring and tracing, unpatched servers, dodgy test environments. To operate, is to test. Solidarity with our friends in operations will go a long way. They have suffered.
• #27: Your own testability is constrained by your dependencies. It has ever been so. I worked on the most beautiful API, acceptance test automation of my dreams, tracing, feature toggles, disposable environments. Apart from a steaming shit of a dependency where we had to raise a jira to add test data. Which would be arbitrarily deleted whenever this internal team fancied a change. Constant build failures. Ended up mocking it, then it was FUBAR in live. Bleh.
• #28: Source control – leading indicator of quality Architecture FFS learn some code – its one of your teams primary artifacts. How can your team get interested in testing if you aren’t interested in THE PRIMARY FUCKING COLLABORATION ARTEFACT. How much is up to you. Build a basic version of your app. Talk about exploration/automation/perf/security/accessibility as though your team should care about it Talk external to your team about things stakeholders care about. Ops people want not to be woken up, Product people want to validate changes. STOP FUCKING TESTING EVERTHING – hold your nerve. Leave it in the column for a day or two. See who blinks.
• #29: Show them the world of testing that is possible. The more testable, the ways become open.
• #30: Everything starts with a leap from someone, it may as well be you. The simplest starting point is to show someone who may be able to help, how you test.
• #31: Has the ability to observe and system has the characteristic of observability
• #32: Has the ability to observe and system has the characteristic of observability
• #34: Being an information provider is usually a position from which we defend ourselves against the largesse of organisations. How can we switch footing from defending to reaching out is the challenge, we need a new
• #35: You know who else was a victim of this. Our friends in Ops
• #36: I believe in testing as a force for focusing on value and risk. For me, this includes less fucking around with features and focusing where it really matters. I believe that testability can enable our teams to share this focus. I believe that it starts with us. Right here, today. All of us. You can do it.
• #37: You will find the principles of testability engineering here