Beyond Usability Testing: Assessing the Usefulness of Your Design
10 likes3,463 views
This document discusses how usability testing can be adapted to assess the usefulness of a design when the goals differ from just finding usability problems. It proposes conducting usability tests with three components: 1) Pre-task questions that set the context of usefulness instead of just demographics, 2) Participant-directed tasks instead of predefined tasks, and 3) Post-task questions that compare expectations and value instead of just satisfaction. This adapted approach leverages the strengths of usability testing while allowing different objectives of understanding usefulness rather than just usability problems.
Beyond Usability Testing: Assessing the Usefulness of Your Design
- 1. Beyond Usability Testing: Assessing the Usefulness of Your Design UPA Boston Mini-Conference 2011 Prepared by: Michael Hawley – Chief Design Officer Daniel Berlin – Experience Research Director May 25, 2011
- 2. Have You Run Usability Tests?
- 3. Usability Testing Participants attempt to complete a set of defined tasks. Researchers learn what to improve by observing and interpreting think-aloud.
- 4. Virzi, R.A., Refining the Test Phase of Usability Evaluation: How Many Subjects is Enough? Human Factors, 1992. 34(4): p. 457-468.
- 5. Trend Business sponsors turn to us as UX professionals with questions that are not about usability problems. Rather, their questions are about overall user experience strategy, value and usefulness.
- 7. Usability: Find interruptions in workflow that prevent users from performing tasks quickly and efficiently.
- 8. Usefulness: Inform a re-structure of the application to best align with workflow. Determine where to position productivity tips and help buttons within the application for best utilization. Find optimal level of personalization and customization that users would take advantage of.
- 10. Usability: Assess effectiveness of navigation system in guiding users to desired pages. Evaluate descriptiveness and clarity of links. Gauge ability of page layouts to orient users to relevant content.
- 11. Usefulness: Identify content that is missing which will help overcome objections or answer critical questions? Understand how branded labels and content themes contribute to the overall experience or detract from it. Determine level/types of promotions and interstitials that are acceptable to users. Understand how different audience personas prefer to consume information for the particular domain.
- 13. Usability: Determine optimal level of difficulty to encourage advancement to multiple levels of the game. Assess discoverability of game features and controls.
- 14. Usefulness: Find the optimal rate of point accumulation and alignment with prize levels. Understand best use of social media within or around the game. Determine the threshold for ads, interstitials and registration for game play.
- 16. Usability: Assess if users can figure out how to add a comment, share content, or use a tagging mechanism to find what they are looking for.
- 17. Usefulness: Determine the most compelling and appealing topics or categories for conversation. Understand the level of involvement the sponsoring company should have in the social experience, if any. Balance branded or non-branded experience for optimal trust of the site. Determine the elements or attributes that should allow comment and review.
- 19. Usability: Find any confusion points or interruptions that prevent users from registering. Find misleading or ambiguous terminology.
- 20. Usefulness: Determine the most persuasive elements that will compel the target audience to register. Understand a design’s impact a user’s perception of the brand. Position the offering and messaging against the company’s competitors. Determine missing content that can help target audience make an informed decision about the product.
- 21. Sound Familiar?
- 22. Natural Reaction Turn to what we know: Usability Testing (one-on-one interviews, design artifact, and tasks)
- 23. Are You Forgetting Contextual Inquiry and Foundational Research? Discovery research and needs analysis is valid, but: • Time and budget for separate research is not always an option • Many participants need design artifacts to elicit appropriate reaction and commentary
- 24. Our Goal Leverage the strengths of usability testing but adjust our approach when objectives differ from finding usability problems.
- 26. Three Components Phase Usability Usefulness Pre-Task Demographics, level of Daily task flow, pain points, Questions expertise, prior experience. expectations, desires, Goal: validating groups scenarios. and classify results. Goal: set mindset for usefulness evaluation of the design Tasks Post-Task Questions
- 27. Three Components Phase Usability Usefulness Pre-Task Demographics, level of Daily task flow, pain points, Questions expertise, prior experience. expectations, desires, Goal: validating groups scenarios. and classify results. Goal: set mindset for usefulness evaluation of the design Tasks Pre-defined tasks, minimal Emphasis on participant- moderator intrusion, directed tasks Goal: find usability Goal: understand how proposed problems, measure time design aligns with user needs on task and completion percentage. Post-Task Questions
- 28. Three Components Phase Usability Usefulness Pre-Task Demographics, level of Daily task flow, pain points, Questions expertise, prior experience. expectations, desires, Goal: validating groups scenarios. and classify results. Goal: set mindset for usefulness evaluation of the design Tasks Pre-defined tasks, minimal Emphasis on participant- moderator intrusion, directed tasks Goal: find usability Goal: understand how proposed problems, measure time design aligns with user needs on task and completion percentage. Post-Task Level of satisfaction with Comparison with expectations Questions the design, and points of and value. Task retrospectives. confusion or ambiguity. Goal: discuss opportunities for Goal: measure usability improvement.
- 29. Example: Usability Goal: Find usability problems Pre-Task Questions: Audience segmentation Tasks: Defined Post-Task Questions: Baseline satisfaction
- 30. Example: Usefulness Goal: Role of personalization Pre-Task Questions: Common tasks, pain points Tasks: Emphasis on participant direction Post-Task Questions: Comparison of expectation
- 31. Summary Foundational research is still important. Usability testing is still important. However, recognize when you have different goals and adapt the research method as necessary.
- 32. Additional Information Complete Presentation Slides • http://www.slideshare.net/hawleymichael • http://www.slideshare.net/banderlin Contact Information Michael Hawley Dan Berlin mhawley@madpow.net dberlin@madpow.net @hawleymichael @banderlin
Editor's Notes
- #5: Goal of usability testing is to find usability problems.
- #7: Transactional Applications
- #8: Transactional Applications
- #9: Transactional Applications
- #10: Content / Informational Sites
- #11: Content / Informational Sites
- #12: Content / Informational Sites
- #13: Gaming
- #14: Gaming
- #15: Gaming
- #16: Social Sites
- #17: Transactional Applications
- #18: Transactional Applications
- #19: Marketing / Persuasive
- #20: Marketing / Persuasive
- #21: Marketing / Persuasive
- #23: So when a client wants discount research to guide their UX strategy and wants to learn about the value and usefulness of a prototype or existing interface… what do we do? We turn to what we know – usability testing.
- #25: What strengths can we take from usability testing?How should the methodology evolve?What questions should we pose to elicit this information from participants?What is an acceptable prototype fidelity level?
- #26: Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
- #27: Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
- #28: Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
- #29: Moderator and participant interact one-on-one (except for unmoderated) Three primary components:Pre-task questions: moderator interviews participant to clarify demographics, assess level of expertise, and gauge prior experienceTasks: participant proceeds through defined set of tasks with design artifact while “thinking aloud”. Moderator minimizes intrusion, identifies usability problems, and tracks metrics such as time on task and completion percentagePost-task questions: moderator asks participant about their experience with the tasks, level of satisfaction with the design, and points of confusion or ambiguityNo longer just determining demographics and level of experienceRather, need to set participant’s mindset for an evaluation of the design from a usefulness perspectiveBorrow from Ethnography, Contextual Inquiry, Interviewing and Laddering Daily task flow, current mechanisms Pain points Expectations, desires Ideal scenarios Other systemsEmphasis changes from contrived tasks to participant-directed tasks Tasks determined from pre-task and probing questions“Tell me how you would use this application in your daily routine. Okay, now please go do some of those on this prototype.”“You mentioned that you would need to determine your size – how would you do that?”Gather expectations at outset of the task “What would you expect to happen? What information would expect the page to contain?”Sets the basis for participant commentary (not that you should design exactly what they say, but it can inform ideas)This may not be a natural way to interact with the design, but we’re not testing usabilityFollow up questions emphasize expectations and value “How did completing this task compare with your expectations? Was it better or worse and why? What’s missing? What’s superfluous?” “How does it compare with other applications or systems?” “What other features or functions would you need when completing this task?”Participant directed to walk back through the task, responding to questions at key interaction pointsStrategic probing questions on salient components that interrupt or inform the task“What would make this more useful? How would you use this in your work/life?”“Does this table contain the columns that you need? Did the categories make sense?” Probe on areas that were not covered by the participant task“How would you get the detailed information about this product?"
- #32: Usability is still important!But, recognize when you have different goals and adjust accordinglyBe aware of nuances and differences between usability and usefulnessSet clear expectations and goals with project teamName it “usefulness study” rather than a “usability test”?Drawbacks: may need deeper prototypes, not getting at usability problems, Danger: asking the participant to help design it is easy to fall into, but not good