Actionable Results from UX Research

Editor's Notes

• #2: Quick bio of each of us?
• #3: Here is just a little info about CarMax the company. We are are coming up on our 25th anniversary pretty soon and will be excited to hit such an important milestone as a company. Our Associates play a huge role in our Product org – not only do we design the customer experience you see when visiting our site or our native apps, but we also do a lot of work on our Associate facing systems as well and they are great at allowing us to test new things in store along side of them.
• #5: As I mentioned I’ve been here 4 years and we’ve been able to assemble a really talented of Product and Visual System Designers. I think it’s important to spend a few minutes discussing how we are organized so that when we get into what good looks like for team norms you have some idea of what a “team” looks like.
• #6: This shot from our office in Downtown Richmond where many of our Product teams are located and we realized that we were continually telling our Product story to new hires, visitors and other teams from within the Company. So we decided to dedicate a wall to telling that story and it explains how we are organized and how we work.
• #7: I’m going to start with the Core Team and what that means. A core team, sometime called a durable team, is a group consisting of a Product Manager, Product Designer and a Lead Developer. These Three Amigos are dedicated to a particular part of the customer or associate journey. I’ll talk a little more about that later. These 3 are co-located so that collaboration and rapid iteration can happen more easily.
• #8: Then surrounding the core team is the whole team. These individuals may support multiple core teams. Those roles include QA, Delivery Managers, Data Analysts, Visual Designers, etc. But when it comes to conducting Discovery – it should ALWAYS included the core team – and any whole team members that are available.
• #9: These core teams sit within one of our Customer Journey Stages. Search / Shop / Buy / Sell. Again they are dedicatde to an opportunity space within a Journey stage but often are required to collaborate and co-design with teams within and across the customer journey. Our VSD team spans across the entire journey.
• #10: So now that you know a little bit about how we are organized – how exactly do we work? We practice what is commonly referred to as “ Dual track Agile” What that means is Discovery is continuous – opportunities, ideas and problems go through the discovery process and make their way into the delivery backlog where the development team can focus on releasing. If you’re doing it right you toss a lot of ideas in the trash, while saving the learning's. What your discovery process looks like will depend on what you are trying to learn and how quickly you can learn it. I’ve been on teams where we go through several rounds of validation in one day and teams it takes longer because of the nature of the problem. That’s your learning velocity, which I’ll talk about next. What DT isn’t – is a product designer going off on their own and handing back leanings to the team. I’ll talk about what good looks like in a second. While a product manager, designer, and senior engineer may lead and orchestrate discovery, they must involve the whole team in discovery tasks wherever possible. Keep discovery work and progress visible to the whole team. Keep measuring and learning even after you ship.
• #11: How does this fit into velocity? Speed, direction? Intro to velocity The UX wisdom of Yogi Berra “You can observe a lot by watching”
• #12: If you are part of an Agile team then you are probably used to hearing the word “velocity” quite a bit. Maybe you review your team’s velocity as part of your sprint planning and retros. Your stakeholders might be asking your delivery managers……
• #13: A predictable delivery velocity is important to the health of the team – but in addition to asking “How much did we deliver?” you should also be asking yourself as a team….
• #14: “How much did we learn?”. This is a total vanity metric – and I would only recommend it to teams who are new to Discovery or who have maybe fallen out of habit of doing continuous discovery. You can start to track and ask yourself, and your team….
• #15: How much research was done as a core team? • Discovery is not the job of the Product Designer to go off an do in isolation – it’s a team sport. • Here we have a Product Designer, Matt and on the right a Lead Developer, Brenda talking to an Associate – I assume the Chris the Product Manager took the pic. Did you GOOB – Get out of the building • We use a lot of remote feedback tools but also spend a good amount of time in the field with our customers and associates. Many of the tools we design should be viewed in the context to how they are used. If it’s something our Associates use in-store or out on the lot or in the business office – we need to see it in context so that we can account for all of the potential environmental factors that play a role in usage or overall usability. • In this photo we have the core team, plus the Field Lead. How Many Experiments Did You Run? • This team tracked on their evidence wall, something Archie will talk about later, the number of tests they ran for that Sprint. Now this is a total vanity metric and really doesn’t mean a thing unless you are getting the proper learnings from it. What it does do is it keeps the customer top of mind for the core team and build good habits that leads to healthy teams.
• #16: So I mentioned doing all of this as a team – but I want to talk about why it’s important to do so. For better or worse, this is the way documents actually work. If you participate in lots of discussions about what software to build, and then create a document to make sense of it, you might share it with someone else who was there. You might both agree it’s good. But remember, your shared understanding is filling in details that aren’t in the document. Another reader who wasn’t there won’t get the same things from it that you will. Even if they say they get it, don’t believe them. Get together and use the document to tell a story the same way I used my vacation photo to tell you my story.
• #17: So at a minimum the core team should all participates in Discovery – when that happens you…
• #19: Method Testing for Value - Would you use it versus could you use it? Testing for value vs testing for usability
• #20: Method Testing for Value - Would you use it versus could you use it? Testing for value vs testing for usability
• #21: Method
• #25: Harvesting insights
• #26: Insights​ This is really important. You don’t need to capture everything. You only need to capture the information relative to your learning agenda. If you record the session, you will never listen to that recording. Facilitator should avoid taking notes. At least one person for note taking but hopefully the entire team can contribute. Difference between what you want to learn and what you need to ask.​
• #27: Instead of using the commonly used tactic of “Imagine you are (insert action). As soon as you say “imagine” you are giving that person liberty to make something up and you are running the risk of not getting an authentic response. This approach might be fine in the “could they use it” category of usability testing when the UI is more what you are seeking feedback on. However when doing early discovery – try instead to get your participants to tell you a story,
• #28: By having customers tell you a story you get to hear the process in their own words and in their own steps. What you assume or think they do is often very different than what they actually do.
• #29: Insights​ Listening - active listening, mirroring, summarizing​ "That's sounds frustrating"​ "What I heard you say …"​ If they ask you "is this right?"​ "so, I hear that's a question in your mind."
• #30: Insights​ Yummy Sounds - behavioral clues, non-verbal expressions of delight
• #32: Insights​ Coding Notes - using a matrix for categorizing notes​
• #33: Insights Coding Notes - using a matrix for categorizing notes
• #34: Insights Syntheses between each session, don’t wait until the end.
• #35: So you’ve collected insights. And you have a pile of stickie notes now what?
• #36: Understanding Synthesizing insights - finding patterns across sessions, Delta-next method
• #37: Understanding Finding pattern from listening sessions and synthesizing into a delta next
• #38: Understanding Delta Next example
• #39: Understanding Finding pattern from listening sessions and synthesizing into a persona
• #40: Understanding You can build your persona within framework that fits your problem space. In this example, the team wanted to align a persona with known data from a broader segmentation study. They also wanted to look at the problem space through the lens of Jobs To Be Done – why would Andy hire CarMax? You can use any framework, the important thing is that you only include insights that are true assumptions and are ”differences that make a difference"
• #41: Added animation to highlight top left Understanding Usability issues - how to track usability issues while testing for value.
• #43: Results Mapping Problem Space - fall in love with the problem, not the solution, sizing and prioritizing Opportunities
• #44: Results Size and proritize Conversations with stakeholders
• #45: Results Crime scene boards no two would be the same
• #46: Summary Bullets