What's Next: Understanding User Research Bias
5 likes1,633 views
The document discusses various biases in user experience research, including researcher bias, respondent bias, and cognitive biases that can impact the accuracy of data collected. It outlines best practices to mitigate these biases, such as framing questions objectively, ensuring diverse sample sizes, and using alternative data analysis methods. The goal is to enhance the accuracy of user research by recognizing and addressing these biases throughout the research process.
What's Next: Understanding User Research Bias
- 1. Powered by What’s Next: Understanding User Research Bias (and Why Your Horoscope is Right)
- 2. Leah Samuelson Senior Experience Researcher Ogilvy Dayoán Daumont Consulting Partner Ogilvy Consulting Welcome
- 3. What’s the weather like in your city? Tell us where you are dialing in from!
- 4. August 20, 2019 4 1 3 4 General bias vs. user experience bias Agenda Understanding bias 2 Overview & best practices Researcher bias Overview & best practices Respondent bias Anything I can answer for you Questions
- 6. August 20, 2019 6 A shortcut our brains take to navigate constant stimulus and make decisions quickly. What is bias? A cognitive bias refers to a systematic illogical thinking pattern that affects judgments and decisions. Biases can impact our actions in positive and negative ways.
- 7. August 20, 2019 7 User experience design is the process of creating products that provide meaningful and relevant experiences to users. What is user experience bias? This involves research throughout the entire process, both in gathering information and testing designs. Biases in UX can cause a respondent to subconsciously respond untruthfully. They can also cause a researcher to evoke a certain response or report inaccurately.
- 9. Confirmation bias Our natural inclination to seek out or pay greater attention to information that supports our beliefs, and discount that which does not.
- 10. 10 “Today you will receive good advice from a friend” DAILY HOROSCOPE
- 11. A human tendency to develop personal opinions, giving more weight to desired outcomes1 May cause the researcher to “seek out” the data that supports their preferences 2 May lead to the observer-expectancy effect where the researcher unknowingly steers the participant 3 August 20, 2019 Confirmation bias Our natural inclination to seek out or pay greater attention to information that supports our beliefs, and discount that which does not.
- 12. August 20, 2019 1 2 3 4Identify any opinions or assumptions you have going into testing 12 Ask yourself which pieces of information you readily accepted, and those you skimmed Make sure your sample size is large enough to provide a greater pool of evidence Write a discussion guide beforehand and stick to it closely Best practices – Confirmation Bias
- 13. Recency bias An order of presentation effect that occurs when more recent information is better remembered and receives greater weight in forming a judgment than earlier-presented information.
- 14. 14 Investors often base market expectations on how it has been performing recently instead of assessing long- term averages INVESTING
- 15. Because it’s easier, we’re inclined to use our recent experience as the baseline for what will happen in the future.1 Studied frequently in social psychology 2 Causes researchers to give more credence to what they heard most recently instead of what they heard most often 3 Can greatly skew data toward the last interview that was performed 4 August 20, 2019 An order of presentation effect that occurs when more recent information is better remembered and receives greater weight in forming a judgment than earlier-presented information. Recency bias
- 16. August 20, 2019 16 2 31 Analyze data in a different order than that of your interview sequence Devise a process for data analysis before testing that gives equal weight to all responses Create a debrief document to fill out with colleagues after each interview Best practices – Recency Bias
- 17. Task-selection bias The effect that takes place when a user knows that a task can be completed, due to the mere prospect of being asked to do it.
- 18. 18 How would you access the virtual dressing room? USABILITY TESTING
- 19. This is a recently defined bias that is rooted more specifically in user experience research1 Researchers in the UX field would not ask a user to do a task unless it was possible to complete, thus clueing the user that the feature is available and findable on the page2 When a user typically interacts with a website they do not assume functionality that is not readily apparent 3 August 20, 2019 Task- selection Bias The effect that takes place when a user knows that a task can be completed, due to the mere prospect of being asked to do it.
- 20. August 20, 2019 20 2 Get general feedback Asking what features a user would hope to have and where they would expect to find them gives helpful feedback without indicating what’s already available. 3 Engage in out-of- the-box activities You can include creativity-prompting exercises such as having respondents draw out or describe their ideal webpage design and functionality. 1 Ask open-ended questions When a user lands on a new page, consider asking, “what stands out to you here?” or “what would you expect to find in this section?” Best practices – Task-selection bias
- 21. Respondent Bias
- 22. Social desirability bias The tendency of respondents to answer questions in a manner that will be viewed favorably by others.
- 23. 23 “I love this design! Delta always has such a creative approach” DELTA AIRLINES
- 24. Social Desirability Bias It can take the form of over-reporting "good behavior" or under-reporting "bad," or undesirable behavior.1 We are evolutionarily predisposed toward this behavior 2 May be caused by social desirability toward others, toward the researcher, or the company doing the testing 3 August 20, 2019 The tendency of respondents to answer questions in a manner that will be viewed favorably by others.
- 25. August 20, 2019 1 2 3 4 Frame questions in an objective manner Exhibit neutral responses and body language Leave the client anonymous until it’s necessary to reveal Administer a survey or unmoderated test 25 Best practices – Social desirability bias
- 26. Anchoring bias The tendency to rely too heavily on the first piece of information offered (the “anchor”) when making decisions
- 27. 27 the initial price offered sets a precedent for all subsequent offers, regardless of market value BUYING A USED CAR
- 28. During decision-making an individual depends on an initial piece of information to make subsequent judgments1 In UX a respondent may often use the first design as an anchor for the following designs 2 A researcher may show multiple designs for a company logo, but the respondent uses the first design as their “gold standard” in feedback for the rest3 August 20, 2019 The tendency to rely too heavily on the first piece of information offered (the “anchor”) when making decisions Anchoring bias
- 29. August 20, 2019 29 2 A/B test designs when possible Showing two or more variants of a page at random is the best means for negating this bias. 3 Deep-dive on customer responses to encourage more thoughtful responses Ask users subsequent questions encouraging them to consider their reasoning. This can also generate feedback on elements that are not anchored. 1 Alternate the order of certain questions from interview to interview Think carefully about the sequence of questions, especially ones that might trigger the anchoring effect. Best practices – Anchoring bias
- 30. Observer effect The observer effect (also known as the Hawthorne effect) is a tendency of some people to work harder and perform better when they are participants in an experiment.
- 31. 31 Productivity improved when the lighting improved… but also when the lighting diminished HAWTHORNE WORKS
- 32. When people are aware that they are being observed, they change their normal behavior unintentionally1 A user my want to complete a task quickly to demonstrate their online savviness or to avoid the discomfort of silence 2 If you ask a participant to add items to an online shopping cart, they may do so faster than normal or triple check the quantity so as not to make a mistake3 The observer effect (also referred to as the Hawthorne effect) is a tendency of some people to work harder and perform better when they are participants in an experiment. August 20, 2019 Observer effect
- 33. August 20, 2019 33 2 Ask warm-up questions Ask the user some light, easy questions to get them comfortable and talking. The more comfortable they are, the less pressure they’ll feel to perform. 3 Encourage users to think out loud In setup and throughout testing, encourage users to engage the think out loud method, providing a stream of consciousness as they work makes responses more natural. 1 Remind the user that this is not a test In the setup of any interview or usability test it’s important to remind users that there are no correct answers. Best practices – Observer effect
- 34. Biases are nearly impossible to avoid. By keeping just a few simple practices in mind, you can improve your skills as a researcher, and improve your data accuracy. BIAS WRAP UP
- 35. Questions?
- 36. Thank you.