Informed & Agile: Test Driven Design w/ Jon Innes

Editor's Notes

• #3: Before we get started, I want to review a few housekeeping items and let you know how you can participate in today´s session. You have the ability to ask questions using your Questions Pane. Simply type in your question and click Send. At the end of the presentation we will do a Q & A session and take as many questions as we have time for.
• #11: Scott Cook founder of Intuit has a great way of putting this…good guidance but it leaves some questions, how do you define big? And what does well mean? In agile we talk about minimum viable products, so this is key. What’s well, or good enough?
• #13: These techniques were identified in the 90’s by various people who all met at a ski resort in Utah 11 years ago to coin the term Agile, and wisely to define a set of values and principles to guide future like minded individuals…
• #14: Note the timelines. You start with basically a to-do list and in the end you’re trying to create something you could ship. Running a traditional test in a month is possible, but 2 weeks is much harder. It can take almost 2 weeks just to recruit users for a traditional study. It means you have to front load some stuff, or run a sprint behind.
• #15: Ever write a usability report nobody read? That’s an example of muda. Did the report note UI inconsistencies due to a lack of communication on the team. That’s mura. Did you stay up all night to write that report. That’s muri.
• #17: Unless you’ve been hiding under a rock, you know that the Lean Startup guys are making a lot of progress here, and their ideas consider UX concepts. I’m excited that Eric and the Lean UX folks have incorporated UX concepts. I’m even more excited that they’ve refocused the agile community on hypothesis testing, which is core to both UX and Agile.
• #18: Janice Fraser coined the term, others like Jeff Gothelf are popularizing it. Today, we’re going to focus on how user testing changes as you become more lean.
• #19: Unless you’ve been hiding under a rock, you know that the Lean Startup guys are making a lot of progress here, and their ideas consider UX concepts. I’m excited that Eric and the Lean UX folks have incorporated UX concepts. I’m even more excited that they’ve refocused the agile community on hypothesis testing, which is core to both UX and Agile.
• #20: Don’t rely on random opinion, or the HiPPO, get facts.
• #21: But we all know that not all ideas are equal.
• #24: Add more attractive images
• #25: Dave McClure, Silicon Valley internet entrepreneur and investor has advised startups to think about 5 key types of metrics using his AARRR acronym. I’ve listed the first 4 here which are UX centric to give you a sense of why UX is not just the product or service design.
• #28: Change only starts when people embrace new values. Tracking objective progress towards the desired outcome by making it visible is key. I’ll assume you care about UX and design, or you wouldn’t be here. The secret sauce in UX is user involvement. The more user involvement throughout the process the more effective your iterations will be.
• #29: Steve Blank author/entrepreneur and business professor
• #31: Here’s an example of a story map from Jeff Patten. Note how this is really just a way of collaboratively organizing the backlog of stories. It focuses on the ideation phase. A great place to start because it focuses the team. It’s great at analyzing user story dependencies and helping teams define the minimum viable functionality
• #33: The UXI Matrix is a tool for tracking UX metrics related to user stories. This is a simplified version showing sample data. It can be extended to cover different metrics or simplified, but the key concept here is to start keeping score, or tracking, how well you are solving an individual story, as well as a set of stories both in a sprint and across sprints. And yes, you could estimate per story for each persona…but that was omitted here to simplify the example.
• #34: The challenge with most traditional usability methods is they are very resource intensive and often take too long for agile timelines. We need to adapt…
• #35: 3x3 is a way of coming up with a conceptual design based on testing early low fidelity prototypes of the alternatives to select the best one. Like Mendel, breeding a better pea. It can save you a lot of rework later doing what Alan Cooper used to call “trying to sand a table into a chair” by ensuring your starting down the right path. It’s ideally suited for new projects or site redesign work. It can be extended to greater numbers of alternatives and more iterations as needed. Key to this type of formative study is recruiting the right participants. Biased results can send you off a cliff.
• #36: RITE or rapid iterative test and evaluation where you run many small sample tests and instead of waiting until the end to fix issues, you fix issues as you go and continue testing. While compatible with agile, scaling over the long term can be challenging as it’s done traditionally. Keep in mind it’s purpose is to rapidly refine an existing design. While it can be adapted it’s best suited for refining a single design.
• #37: A/B tests are very efficient at refining final details such as button size, color or placement for an existing site. But back to what Alan Cooper said, you can’t sand a table into a chair. A/B testing is well not suited to fixing major design problems. It’s a power sander. If you rely solely on intercept testing, you may never expand your market beyond the early adopters who might be giving you the wrong feedback for your ideal market.
• #39: Note the differences between this an A/B testing.
• #41: With large scale automation comes the potential for large scale error. Always pilot, or you’ll end up paying participants for nothing. Remember that you could be introducing a bias if you only test with users who respond to your requests in a very short time. Take advantage of the low cost of intercept recruiting when refining for existing users, and save the cost of panels for what they are best suited for. Avoid trying to run 2 hour studies, you’ll end up with lots of drop outs. Beware of non-automated methods that could create lots of work, plan for open ended questionnaires or other analyses that might be bigger than you bargained for.
• #42: There are many things that you can front load to make your life easier. It’s like having gas in the car before driving to work. There’s nothing non-agile about that. Take personas. If you work with the product owner or marketing staff to define these, it makes recruiting much easier. The budgets for recruiting from panels should come from marketing budgets. Get the whole team involved in defining tests and metrics, including setting goals for releases to improve metrics. Stop thinking about user testing is something you do at the end. That mindset is an artifact of waterfall development.