Fast and Unbiased Usability Research
2 likes642 views
The document discusses methods for conducting fast and unbiased usability research. It addresses reducing bias through careful planning, structuring studies, focusing on priorities, automating tasks, and reducing cognitive load. It provides recommendations for efficient participant recruiting, study design, data collection, analysis, and reporting. The goal is to produce high-quality insights and actionable recommendations in an optimized manner.
![@kaniasty | #uxpabos17
18
The Session Summary
This participant had major issues with X and Y features
because she expected [X] and [Y] to happen. She followed an
interesting workflow that was different from what we
expected. She also did [this interesting or surprising thing].
She was hesitant to explore the UI and fit the ‘Ms. X’ persona
pretty well…
“Take 5.”](https://image.slidesharecdn.com/fastefficientresearch-170522173109/85/Fast-and-Unbiased-Usability-Research-18-320.jpg)
Fast and Unbiased Usability Research
- 1. @kaniasty | #uxpabos17 You can have it both ways Fast and Unbiased Usability Research Eva Kaniasty, Red Pill UX
- 2. @kaniasty | #uxpabos17 2 What do we want? Everything. When do we want it? Now
- 3. @kaniasty | #uxpabos17 3 What is good research? “You can’t always get what you want.”
- 4. @kaniasty | #uxpabos17 4 Bias is everywhere! • Inter-evaluator bias • Loaded or leading questions • Cognitive biases (e.g. primacy, recency) • Group dynamics (i.e. groupthink) • Prejudgements • Selection bias • Respondent bias (e.g. positive, social) “Soylent Green… Bias… it’s made of people!”
- 5. @kaniasty | #uxpabos17 5 • Plan • Structure • Focus • Automate • Don’t rely on memory Reduce Your Cognitive Load “An ounce of prevention is worth a pound of cure.” - Ben Franklin
- 6. @kaniasty | #uxpabos17 6 “Efficiency is intelligent laziness.” Objectivity through Optimization
- 7. @kaniasty | #uxpabos17 Good + Efficient Participant Recruiting
- 8. @kaniasty | #uxpabos17 8 Recruiting Recruiting Brief Screening Survey Scheduling - Research Recruiter - UserInterviews.com - Stakeholders Participant Data - Contact Info - Time Zones - Screening Data “Keep it simple…”
- 9. @kaniasty | #uxpabos17 9 Customer Super Screener (i.e. survey) Better screening - Larger pool of potential participants - Foot in the door - Comparison sample - Bias fighter Measure twice, cut once Collect data as you screen - Demographics - Feature usage - Card sorts - Desirability (product reaction) - Technical - Quotes (collected judiciously) - Opt-in for future research (online & moderated)*Don’t forget legal review
- 10. @kaniasty | #uxpabos17 10 Recruiting Logistics Written Consent •Context •Logistics •Recording •Compensation •(Optional) Nondisclosure *Don’t forget legal review “80 percent of success is just showing up.” Reducing No-Shows •Written consent •Email confirmation •Remote tool testing/setup •Phone/Texting Session Status Tracking Compensation
- 11. @kaniasty | #uxpabos17 Good + Efficient Study Design
- 12. @kaniasty | #uxpabos17 12 “Get your priorities straight.”
- 13. @kaniasty | #uxpabos17 13 Moderator Guide Pre-Test Interview - Background / Profile Fit - Past experience/usage - Top 3 tasks/needs First Impressions - (if website) HP overview - (if navigation) Navigation expectations - (if new user) Free exploration - (if current user) Recent task Tasks or Process - Probing questions - Ease of use ratings - Task Success Wrap-Up - Salient Likes/Dislikes - “Change one thing” - System Usability Scale/Global Measures “How the sausage gets made.”
- 14. @kaniasty | #uxpabos17 Good & Efficient Data Collection
- 15. @kaniasty | #uxpabos17 15 Session Housekeeping Participant Briefing - Context-setting - Recording consent - Think-aloud instruction - Honest Feedback - Neutral Observer Logistics - Apps / Links - Notes templates - Equipment - Logins - Recording - Reset “Don’t forget to eat.” - Mom
- 16. @kaniasty | #uxpabos17 16 Recording & Photography • Personal room • Test meeting OR Tech call • Mac & Windows compatible Mobile Screen Mobile Device • Sleds
- 17. @kaniasty | #uxpabos17 17 Super Notes! Mod Guide + Notes + Audio PearNote Mac OneNote Windows Livescribe
- 18. @kaniasty | #uxpabos17 18 The Session Summary This participant had major issues with X and Y features because she expected [X] and [Y] to happen. She followed an interesting workflow that was different from what we expected. She also did [this interesting or surprising thing]. She was hesitant to explore the UI and fit the ‘Ms. X’ persona pretty well… “Take 5.”
- 19. @kaniasty | #uxpabos17 19 Avoiding the Slow TV Syndrome • Re-watching videos • Relying on notes alone • Human transcription • Automatic transcription VLC player
- 20. @kaniasty | #uxpabos17 20 Task Data
- 21. @kaniasty | #uxpabos17 21 Usability Measures Custom Tools Question Banks
- 22. @kaniasty | #uxpabos17 22 Question Banks Product Reaction CardsSemantic Differentials Ease of Use
- 24. @kaniasty | #uxpabos17 Good & Efficient Reporting
- 25. @kaniasty | #uxpabos17 25 Analysis & Reporting Strategy Findings Report - Narrative - Includes detailed study methodology & background - Includes raw data - Includes participant detail - Includes session summaries - Focused on detailed findings Strategic Brief Presentation - Focused on the big picture - Focused on priorities - Focused on critical findings - Includes recommendations - Visually impactful - Includes quotes - Synthesizes data from multiple sources - Summary deliverables: (e.g. profiles, personas, user journeys) “Use the right tool for the job.”
- 26. @kaniasty | #uxpabos17 26 NOT Survey Reports No Custom Color Coding Always reports %
- 27. @kaniasty | #uxpabos17 Participant Data 27 ♀ 5 Female ♂ 5 Male 4 on Laptops 4 on Smartphones (4 iPhone, 3 Android, 1 Windows Phone) Customers Company 1, Company 2, Company 3 Customers RepresentedProfile 1 | Profile 2 Session Summaries as part of detailed findings / or with recordings
- 28. @kaniasty | #uxpabos17 28 Task Data
- 29. @kaniasty | #uxpabos17 29 Structured Data
- 30. @kaniasty | #uxpabos17 30 Usability Measures Tullis & Albert
- 31. @kaniasty | #uxpabos17 31 Data Analysis
- 32. @kaniasty | #uxpabos17 32 Infogr.am PROS: Range of Graphs Fun viz options Full control over colors Option to show values (not %) CONS: Limited font control No option to show confidence intervals
- 34. @kaniasty | #uxpabos17 34 “Last but not least… qualitative data analysis.”
- 35. @kaniasty | #uxpabos17 35 Final Presentation Strategic Brief - Focused on the big picture - Focused on priorities - Focused on critical findings - Includes recommendations - Visually impactful - Includes quotes - Synthesizes data from multiple sources - Summary deliverables: (e.g. profiles, personas, user journeys) “Looks matter”