Joanne White - Research for Every Company: How to Build Lean, Agile User Research Participant Forums

Editor's Notes

• #2: Good afternoon, it’s great to be here. My name is Jo, and I’m the Lead UX Researcher at Willis Towers Watson, which is a global business consulting firm. If you haven’t heard of Willis Towers Watson before, we’re like Price Waterhouse Coopers, except we wouldn’t have stuffed up the envelopes at the Oscars. We consult to Fortune 500 firms around the world on myriad aspects of human resources. We have six proprietary software and data products that are used in that consulting, and which are also sold directly into those firms to have in house. My role includes managing the research that supports the development of those software and data products.
• #3: This talk has a few different parts. So to kick us off, you can get to know me a little, and get past my accent a bit, by doing what I like to do best - telling you a personal story. Now pay attention, because there’s a quiz. (An easy one)
• #4: This is me and my mum, Susan. My mum was a bit of a handful when she was young. When she was just 18 years old, she moved to Australia from England, on her own. She met my dad in Australia, and together they built a gorgeous family of their own. Mum always had a very strong sense of family, and she wrote to her mum, my grandmother, and her three sisters every week. As time went on and international phone calls became a little less expensive, they would call regularly. Finally, when social computing finally met the broader consumer market, mum and her sisters would get together on Skype every Sunday evening at 7pm for an hour-long chat. This weekly catchup happened for years. The charm of this is becoming lost as we become more used to free global communication, but I’m hopeful some of you remember those olden times. Anyway… it was and still is less expensive to use technology than to fly to see each other, but one year, grandma and her sisters planned a holiday where they would fly to Australia to visit my mum and dad. They planned it in the July for the following January, so six months in advance. Tickets booked, the lot. My mum was so excited. She talked about them coming out constantly.
• #5: A couple of months later, Christmas was coming up, and I was stuck with the ‘what on earth do you buy for your parents’ question. We’ve all been there. Stewing over getting a good gift. Wandering aimlessly through shopping centers. You just don’t know what to get, but you have to get ‘something’. And then it hit me.
• #6: I stopped thinking of my need to get a gift. Instead, I thought of my mum and what was going on in her life. I put myself in her shoes — and I realized something. I remembered that my mum, my grandma and mum’s sisters all liked a glass of sherry of an evening. Mum had mentioned it in a couple of stories she’d told me. Opportunity! I bought her a set of sherry glasses so when grandma and mum’s sisters came to visit, they’d be able to sit together each evening and have a glass of sherry.
• #7: Win! Don’t you feel good when you just know you got the perfect gift? And isn’t that gift perfect because you know it reflects the person in a way that’s hard to describe? Okay, I’ll finish this story later, but it brings up a few good points to lead in to what I want to share with you today.
• #8: Now for the quiz: What was my problem in this story?
• #9: If you said I needed to get a gift for my mother for Christmas, you’d be right. But that’s not enough. It’s a really basic understanding of the problem. It doesn’t appreciate the depth and parameters of the problem beyond something very functional - get a gift, by December 25. Thinking of the problem that way is useful in that it’s broad and translatable - you can imagine the same kind of issue if the customer were buying a gift for someone else, or for a different occasion. There are lots of similarities which can be helpful. But there’s a danger in only appreciating the problem at this level.
• #10: It’s impersonal. In it’s generalizability, we are starting to lose touch with the customer’s context. If we only define the problem that way, we might be able to solve the problem, but we’d likely have a hard time ‘solving it well’ from the customer’s perspective. The customer’s experience is different depending upon the social influences and other aspects of context going on. Even if it’s a daily purchase, or an annual occasion like Christmas, if we are to get our customer experience right, we have to recognize and include and appreciation of that social context. Often it’s that social context that will make a huge difference.
• #11: Coming up with the idea of sherry glasses relied on more than a surface understanding of the problem. Centering on my mum’s contextual reality, led me to feel empathy for her, and that led to seeing the opportunity that would resonate best for her. If we can’t empathize with customers by understanding their experience in their own context, then we are not being open to really being open to solving their problems well. So let’s talk a little more about empathy and what it looks like.
• #12: Having empathy for your customer adds rich detail to your understanding of who they are, their behaviors and attitudes, and their experience of their problems, beyond a dataset of demographics. This understanding is gained through mixing both quantitative research and qualitative data. The mixed method approach gives you general trends as well as nuanced insight that helps indicate how your customer sees the problems they’re experiencing. In gathering this data, you can use your own experience to help you empathize with someone, but ultimately, the practice of empathy means you need to walk in their shoes - and that’s a heavy thing to do.
• #13: When you conduct research from a center of empathy, it influences how you ask questions. This is relevant across both quantitative and qualitative methods. If it’s a survey, the questions served up will reflect what answer the participant selected beforehand, to best get data that reflects their individual experience. Digital survey tools support this kind of piping - you just need to understand that some respondents are likely to go different ways to others, and construct your surveys to reflect them. If it’s an interview, you’ll work from an interview guide rather than a checklist of questions, and adjust on the fly depending upon what the participant seems to focus on as being most important to them. Being empathetic in interviews means you are going to want to audio tape it rather than take notes, which are often distracting to both you and the respondent. You’ll identify whether something is appropriate to say or do (or just as importantly, whether something is NOT appropriate to say or do). You’ll use words that they use. You’ll ask better questions that make the respondent feel more comfortable. This situated research is about building a relationship with the participant. Customer centered research plans that use this approach of empathy will readily reflect the most ethical research practice because you’re basically going to treat the participant the way you’d want to be treated.
• #14: Sure, we all have pressures and stakeholders with different interests - launch dates, product managers, sales departments. We deal in spreadsheets, offices and boardrooms. Our product’s logo, messaging and mission are focusing everything we do. But at the end of the day, the only reason any of these things exist is because of the customer. It can be easy to assume we know our customers because we have a survey, and we spend all day thinking about the product or service we deliver to them. But that’s not really knowing the customer. It's knowing ourselves.
• #15: Customers, and their problem spaces, are not static. It doesn’t matter if you’re in B2B or B2C; whether you’re supporting a product that hasn’t changed in many years or whether you’re working on a startup. You need to not only do research, but do it constantly, hand in hand with your customers. And that research has to focus on the problem as well as your solution, at all stages of the product lifecycle and all stages of your customer relationship.
• #16: I know most of you are thinking, oh we do research. I know this. We do surveys; we do email campaigns for feedback. I’m going to tell you that I bet a lot of you are missing the target with your research, and I’m going to tell you how you can start to fix it.
• #17: The thing is, research in our industries has become standardized around two things. And these two things are hurting you. First, research has been limited by the method or methods that are familiar to you and the customer (even if everyone finds it boring); and second, research is limited by how convenient it is for you to get access to customers. The lack of caring on all sides in all this is palpable. You don’t care if a survey is going to get you everything you need to know about a customer’s needs - you just want the data. In fact, you just want data that confirms or denies things you already know within a scope that you’re okay to deal with. And you know your customer doesn’t really want to fill out yet another survey because heck, you personally don’t fill them in for other businesses, so why would they? That’s why you sweeten the pot with gift cards, loyalty points or raffle draws. Research has become an inconvenience rather than an opportunity.
• #18: Let me tell you why not questioning our research methods is a problem. In his book Streetlights and Shadows, Gary Klein tells a story about a policeman who is helping a drunk man look for his keys under a streetlight. After a while, not finding them, the policeman asks the man where exactly did you lose them? The drunk man points towards a dark alley and says “over there.” The policeman asks “so why aren’t you looking over there?” And the drunk man replies “because the light is so much brighter over here.” We are guilty of looking for ways and times of finding information that suit us rather than those that suit our customers.
• #19: Research is inconvenient because it is hard to find the time or authorization to access customers in both B2C and B2B environments. Guerilla research is often recommended to ‘solve’ that problem. You might pay an agency for them to recruit a number of typical customers to engage with. Or you might visit with people in a coffee shop - ask them to try out a user interface or give an opinion on something, and in return you buy their coffee. Cheap, easy, quick data? Sure. But guerrilla research gives you a single opportunity to speak with that customer, and think about it - if you need to pay someone to give you an opinion, what is their care factor really? Guerrilla research is like the fast food of research - it’s okay every now and again, but it won’t sustain you long term.
• #20: Just like the drunk in the story who was looking for his keys, we rely on methods of research that are convenient to us, even if it’s not going to get the best results. We send thousands of generic surveys on behalf of our companies, ‘asking for help’ about how the experience was, but we’re really not interested enough to personalize it. We ask people after they’ve left the building physically and mentally, for their opinion. Rarely do we collect data on the context of their experience. We don’t refer to the time they connected with us, what they bought, who they were with, how often they’ve been in this month… what message do you think that sends?
• #21: I wonder if in addition to time and access, it’s because we fear we might not like what we’re going to hear? It’s not easy or comfortable to be in a problem space with a user/customer, especially if your solution is not fixing their problem! But that’s when you need to spend even more time with them, to better explore their problem space with them, find out how your solution can be better, and being open to including their voices as you report back to stakeholders and work together to improve. If you are open to customers and their problem space throughout their relationship with you, they will be honest and confident in sharing their view with you. And it is through creating customer forums that you can build a research relationship with people you will get to know, who will tell you the truth, and who will feel valued by you in the process.
• #22: So how do you go about building Customer Research Forums, who goes in them, and how do you use them?
• #23: In building your forums, you want to find people who are really interested in the problem you’re working on. They might not be a customer you already have, they might not have heard of your brand before, or they might hate your brand. Either way, they have a real passion for the problem you’re working on. Think about it - we all have something that we would love to give our opinion on if someone really wanted to know. It might be something that you would never guess, too. Like online banking, or using car tyre pressure pumps and meters at the gas station, or even how to navigate the TSA. We all have at least one thing we feel we are expert in. The great thing about these people is that if you practice good research techniques and engagements with them, they will love to give you their passionate, informed, interested opinion, and it won’t cost you a cent. You might build different forums of participants for different purposes - existing individual product areas, and ones in development, for example.
• #24: Work with your Product Manager and other stakeholders to strategically identify the kind of people you want in your research forums. Demographics and personas are a great start - you want to have different attitudes, and behaviors represented. You want to have the end user, the buyer, any influencers all represented. It won’t be long and you’ll have different stakeholders asking you to add more customers to the research forum. Customers at risk or customers who are really interested in a particular area - all are good to approach and include.
• #25: When you invite someone to join your feedback forum, give them an overview of why you’ve selected them and what the focus of the Research Forum is. I conduct a short introductory discovery interview with them to find out their particular areas of interest around the problem. If the problem space is one that is shared, such as an enterprise B2B environment, do the interview with the team of people being invited. This will let you discover how they like to deliver feedback, and how their context of use of your product or service fits with their usual work environment. The number of people in a feedback forum depends upon your capacity to support engagements with them. Quarterly contacts seems to work well as a standard, and you’ll want to track these engagements in a central CRM or similar tool.
• #26: Now you have your participants, you can figure what kinds of methods to use with them. Remember, some people prefer to give feedback in different ways than others and it’s smart to use them in formats that play to their strengths. Some like to talk on the phone, some like face to face group activity, others might prefer to create snapchat stories for you of their experiences. You’ll get better results if you encourage their preferred method. Having regular engagements with participants allows you to build an ethnographic understanding over time, along with a deeper relationship and it allows you to more easily trip over information that nobody felt was relevant, until it is.
• #27: The way you conduct your research with your forum participants reflects your brand’s image, and it’s personality. Every touchpoint you have with a customer must reflect your brand. Why is it we have scripts and train staff in exactly how to address customers, yet we throw a nondescript branded survey at that same customer after they’ve departed? Not only that, but you have an ethical responsibility to not waste people’s time and to conduct efficient, effective research with reliable results. When people feel respected and valued, they move to being actively loyal brand ambassadors. It doesn’t matter if you’re in a B2B or a B2C space, if someone is actively engaged in a relationship with your brand that they feel positive about, then they will smile when they hear your brand’s name. They will tell others about their touchpoint with you. We recognize this about so much of what we do, so it’s time to incorporate this in our research as well. Research needs to be memorable. That’s your challenge.
• #28: While both quant and qual have limitations, combining different methods can both show trends and provide deeper insight to customer behavior at every stage. Combining the results of research from both is called Thick Data. It’s where you can use quant methods to identify trends - the ‘what’ - and then use qual research methods to drill into those trends and find out the ‘why’. Combining the two types of research in a situated manner with known participants in your Research forum means you can combine and innovate the research methods in different mixes which over time gives you ethnographic insight. Mixing methods like surveys and web analytics with data gained from focus groups, interviews and observations provides deep insight and data in formats that make reporting to stakeholders more interesting.
• #29: But research is about more than method selection. It’s about knowing what you want to find out. Here we’re borrowing from Sprint Research Design, identifying three categories of research areas you can use as a base. Strategic research activity informs questions around long range direction and opportunity. Tactical research is focused on upcoming features and enhancements to your product or service. Validation is reflective and helps ensure what you’re delivering is effective.
• #30: All three of these types of research are done in tandem, but the timing of the research activities and delivery of findings to stakeholders are a little different. You can see the strategic activity is most lengthy, the tactical engagements are more targeted with deliverables, and the validation engagements are the most frequent, with focused check ins and shorter spans.
• #31: The ongoing administration of your feedback forums is worth investing in, even if you’re working in a fast, lean environment. You’re actually front-loading a good amount of work that will reap benefits for stakeholders in ease of access and revisiting them for different activities.
• #32: Working in a team or startup focused on fast deployment means you’re pushed to deliver everything fast. We fall into a dangerous space of doing minimal discovery research and then being ‘finished’ with that, we put it away and focus all our research on the solution.
• #33: While you’re focused on your solution; a competitor will come in and bring a different lens to the problem, which will disrupt the space. Now, your competitor might not actually be offering a better product, it might just have that something shiny factor which draws your customers’ attention. But whether it’s a better product or not, your ability to pivot is limited because you just don’t know the problem space well enough to know where your opportunity might have shifted to. That’s why you want to continue research activity across all three of those areas, your strategic work on the problem space, the tactical work on the solution, and then the validation work to provide a check on both.
• #34: Now back to managing your research participants. People like having interesting ways to be part of research and feedback. If you’re telling them they’re part of a special research forum group, make sure you treat them that way. Show their value by seeking their opinion in ways that reflect their knowledge, experience and preferences. If your participants are expecting to be contacted quarterly, it’s up to you to ensure that happens. And no matter how good your research activities are, and how special they feel, for all kinds of reasons, participants will eventually move on. Be ready to work with stakeholders to identify new participants to add to your forum over time.
• #35: You might not yet be ready to initiate customer research forums, but the need to attend to customer, context-centered research to inform your organization is one that is vital. Research is not an inconvenience, it’s a necessity. And most are wasting the opportunity.
• #36: To review, the number of responses is important, but so is knowing who is giving it, and what context they are experiencing your product or service in. Revisiting with participants in a forum makes them feel valued and gives you ethnographic insight over time. Making your research engagements with participants interesting, unusual and personal makes them memorable and builds brand loyalty. So to finish, let me revisit the story I kicked off with, about my mum and the sherry glasses at Christmas.
• #37: Getting those sherry glasses was a great gift, and would have been lovely on their own. But I took it a step further. I made it more personal. I had my mum, her mum and her sisters’ names etched on each of the glasses. This meant they could each enjoy their glass while on holiday in Australia, and then they could take their own home as a keepsake too, which they all did. With a little thought and ongoing ethnographic insight on your side, it’s so easy to be personal. Thank you.