Change the conversation keynote StarWest 2015

Editor's Notes

• #2: I want you to think about your last release to production – to your customers. Was it successful? <show of hands>
• #3: The conversation I am talking about is the conversation on quality How many bugs …. What kind of bugs… and so on. Only matters when there are too many to be able to use the application And that’s really not what is important to the customer – in today’s market, if you are paying for something, you expect a quality product. What’s important to your customer?
• #4: We measure … What type of bugs, What severity, what priority We have defect triage – estimating how long to fix Waste
• #5: Sometimes we measure other things, like ‘did we get all our requirements in, and did we test them?” How important is this? In today’s age of agile, a disciplined delivery team works at a feature / story together with the customer, who accepts. We know we test all the requirements because we do it together. There’s not usually a need to add extra work to trace. If necessary, we can extract the tests and test results from the automation for audits. There are exceptions, but I don’t come across them very often.
• #6: Measuring anything gives them - management a feeling of control, but it’s an illusion. We can’t control.
• #7: Probably not much … or as much as you think. Much of the metrics we use, probably doesn’t‘ even tell us about the quality of the product, but more about the quality of the process. Does the customer care about the process or the product. Delivery teams often get stuck in the details.
• #8: Value – what is it? Uncertainty – how can we reduce it? Risk – how do we mitigate it? How might we really measure product quality
• #9: Carcassonne, France – medieval city Can we deliver what they want …. Or better yet, what they need?
• #10: value is not the same for everyone When we understand what problem the customer really has, maybe we’ll stand a chance of actually delivering what they need.
• #11: So instead of specifying # of steps, the rise of each step, etc. Maybe we start with “We need to be able to escape from the floor we are on, and get either up to the roof, or down to the street”. When we build it, we have a common understanding of the problem we are trying to solve, when we test it, we can test to the problem & solution, not only to the specification.
• #12: When we think about business value, we need to think about priorities. Priorities… the most valuable first Let’s think about it.. Can we break a feature into smaller bits and deliver “just enough” Story done Feature done ... Work with customers to understand the importance of acceptance. Try to get small acceptance of features – define feature acceptance, not only story acceptance
• #13: I realized that I had done the pieces I really wanted to .. The dress, the pajamas, those are really boring – lots of white, and not much value to me.
• #14: Think “outside the box” – it doesn’t have to be major, and sometimes one new idea makes a process a whole lot easier. The most successful teams enjoy a culture of learning where everyone on the team is free to raise issues and experiment. Experiment to change how you think about measuring value. Validate early – Customer acceptance – on each story, on each feature. Reduce the risk of getting it wrong.
• #16: Project managers often want certainty, management wants control – or the illusion of control But perhaps, we shouldn’t aim for control, but think about how to reduce uncertainty –
• #17: Introduce the Cynefin model so people can go explore and learn. Cynefin Framework (pronounced Ki neh’ vin) is about different domains/contexts -------a sense making model (Dave Snowden Simple (obvious/known)– we’ve done it before; example – communication protocols Complicated – harder, but not new Complex – unknown Chaotic – reacting, responding to fires
• #18: Models can help us choose how to tackle a problem … Is it simple – we’ve done it before : the same, no need for analysis Complicated – harder, but not new - can apply good practices like break into smaller chunks, requires expertise, good practices like ATDD or BDD Complex – unknown - needs probing, questioning, works through examples to understand, experiment, get a better understanding of what it means before we commit Chaotic – reacting, responding to fires - couldn’t treat each problem exactly the same. We don’t want to do the same thing for every single new feature that comes up. We need to think about how to reduce the uncertainty in those complex problem areas. But not waste time on known problem areas.
• #19: Do we have danger in our s/w development - Especially if we are working on a life critical application. Comes back to context.
• #20: Example of project risk – scope, schedule, budget Example of product risk – features, quality attributes such as performance From a testing perspective, we usually mean the second – risks to the product.
• #21: In Dan North’s keynote at StarEast, he talked about several things including the idea of risk planes. One of his Software Faster patterns - These are his pictures – used with his permission. Risk planes – 3 dimensions: likelihood, impact and probability. D – high impact, high likelihood should probably have higher coverage … F – unlikely to fail, and if it does fail, do we care? 80% too much testing We call this risk based testing,
• #22: Components work together, not in isolation. User journeys…. Weakest link
• #23: Each context belongs to a human being Testers need to be inside each of the stakeholders heads, and ask… We go back to the value to the customer, what is their risk threshold – what can they live with.
• #24: So, I said there 2 risks – but that is a myth … There are many stakeholders, many risks. Their perceived risk may not be what you are thinking at all. By asking why they want you to test, perhaps you can figure out what the risks are they are trying to mitigate – what is the most important thing? Ask not only what’s the best, but what’s the worst thing that can happen. Unnecessary features add unnecessary risks. Story about Australia – study showed that speed and alcohol were not the only major causes of accidents. It turns out that tiredness is right up there as well, so they addressed that issue by putting reminder signs like this along the way, and made sure there were rest stops on the major highways. The accidents were reduced. Started in 1986 How do we make sure our team and our stakeholders get the same understanding.
• #25: Risk is reduced when more people know the issues … It’s about transparency – Make it easy for people to participate in helping …
• #26: Use brain storming techniques such as mind mapping This is about testing early - Think about prevent defects instead of counting them after. How do you know when you prevented a defect? By the ah ha someone says. But that’s harder to prove… harder to count. When we make testing visible and reducing misinterpretations, and misunderstanding – it helps to building the right thing. Think about the business rules and explore examples - at all levels
• #27: We often get engrossed in the details – that’s where our expertise comes in, but we need to think about the bigger picture. We need to step back a bit and look at where will it grow, how big will it get, do I need …..
• #29: I liked the way Matt Mansell really says Quality is everyone’s job. A contributor the solution – much better than the person who breaks the software… Holistic view – everyone’s responsibility ;
• #30: Do you look at your problems and really understand them. I see so many teams that aren’t addressing real problems, or don’t understand some basic ideas.
• #31: Customer validation and experimentation needs to continue throughout product/solution development - from the initial seed of an idea through to feature enhancements of a mature product/solution. I like to think of it in terms – - Problem validation, - making sure we understand the problem we are trying to solve - solution (definition) validation – making sure we have a shared understanding of what we are trying to build – testing assumptions early - implementation validation…(testing the product) Our attention should be on solution-market fit versus number of bugs we found. Right now, if feels like we are often victims of our rituals – what we know, what we are comfortable with.
• #32: They are good things. Is it enough? If you aim for zero tolerance during development - zero known bugs escaping an iteration (or story or feature), then we can look very closely at the issues found by the customer because there are few. What about the business value the software adds? What about revenue generated by the the software added? What about the cycle time of work? --- again – that’s measuring process quality – not product quality.
• #33: Monitor how your customers use your product. How is the customer impacted in real time so we can act quickly.
• #34: If we address all the high risk items and only low risk items at the end of the project tells you what? For example, if one of the biggest risks is usability, perhaps we can use some testing techniques borrowed from the user experience world – techniques like customer touch points
• #35: Use different approaches and tools Ex. Story mapping, Impact mapping, Structured conversations
• #37: How can we become more flexible in our thinking – in getting better at what we give to our customers. Just imagine, putting out a release with very few known defects … and being able to measure customer satisfaction not by bugs, but by real value. We know that you get what you measure – think what is important to your organization. Stop, and look at what you are measuring. Does it add value to the organization? Is there something else that we could measure, or change. If you stop counting your defects, (which includes, logging them all and triaging), how much time would that give you to start thinking about the business problem, and being able to participate in validating the solution to that problem before the code is even written. Knowing how many bugs are in your software tells you … how bad it is, and I’m pretty sure that is not what organization thinks is important. We need to create a good / create impact on the minds of the customers. That's is what will give loyalty to your product.
• #38: 37