Beyond usability testing: Getting a holistic view of the user experience
Download as PPTX, PDF1 like1,710 views
This document provides an overview of qualitative user research methods beyond traditional usability testing that can provide a holistic view of the user experience. These include field visits to observe users in their natural environment, informal interviews via phone or email with screen sharing, live recruiting tests to observe more natural user behavior, surveys to collect quantitative data, and diary studies where users record activities and issues over time. The document discusses when each method is appropriate to use and provides tips for implementation. It emphasizes starting research quickly with minimal resources and getting stakeholders involved.
Beyond usability testing: Getting a holistic view of the user experience
- 1. + Beyond usability testing Getting a holistic view of the user experience Amanda Nance, User experience researcher Atlanta Web Design Group Aug. 29, 2013
- 2. + About me In UX field since 2002 Studied human-centered communication & psychology, Mercer University M.S. in Human-computer interaction, Georgia Tech Women’s barbershop quartet @amandaux #beyondutest
- 3. + Context Large, international corporation (14,000 employees) Accounting products for small businesses Desktop and web-based No analytics My product Product Management team - 5 Developers - 16 Dev Operations - 5 QA - 6 Interaction & visual designers - 3 Writers - 2 part-time Researcher - 1
- 4. + Context 31 studies since Jan 2010 120+ users
- 5. + About you
- 6. + My goal Broaden your toolkit for including users Convince you to start right away
- 7. + Usability testing is great. Try it! See “Rocket Surgery Made Easy” by Steve Krug
- 8. + An ideal usability test would tell you everything, but… Ideal In practice Real users yes! Identify unknown needs Natural environment User's own data Real motivation debatable Non-core tasks Major usability problems apparent yes! Minor usability problems apparent yes! Quantitative information sometimes
- 9. + So we have these other methods
- 10. + Field visits
- 11. + Watch and interview users in their natural habit
- 12. + Use field visits before designing to learn what people need Before designing can mean: Before anything Before finalizing the design After launch to improve product
- 13. + Asking users about problems limits the problems you’ll uncover Used under CC license Leah Gregg
- 14. + The problem we solved was one users hadn’t thought about “I hadn’t even thought that far out of the box.” -User reaction to our product concept
- 15. + Observation gives you details about what people need
- 16. + Use field visits for learning about sensitive topics
- 17. + Tips: Before the visit Ask about pets Talk to the participant yourself before the visit Ask to save up some work Even if using an external recruiter Tell them not to clean up! Ask for driving directions / verify Google Maps accuracy
- 18. + Tips: During the visit Say “Show me” and ask for specific examples
- 19. + Tips: During the visit Verify your understanding in the moment Always respond professionally
- 20. + Field visits: general guidelines We pay $300 for 3 hours Limit is 3 hours Can divide into one-hour sessions amongst 3 users
- 21. + Informal interviews Calls & emails, screen sharing
- 22. + Use informal interviews for (perceived) requirements Email and phone calls: easy and quick!
- 23. + Use screen sharing interviews to learn how the product is (really) used We didn’t account for users who repeatedly subcontract for the same company.
- 24. + Screen sharing tips: recruiting Find customers who are interested in the topic Analytics Survey responses Email Explain logistics: ear buds or speakerphone required Get participant’s phone number
- 25. + Screen sharing tips: tools Join.Me - recommended by colleague
- 26. + Screen sharing tips: tools Turn off the beeps when people join Turn off chat for observers Turn off list of meeting attendees
- 27. + Screen sharing tips: during the session “Can you show me…” the issues you’re having how you _______ Help with issues at end In case of emergency: let participant control your screen
- 28. + Screen sharing: general guidelines We pay $75/hour $25 for 20-30 min Limit to 1 hour
- 29. + Live recruiting test AKA intercept test
- 30. + Live recruit test = watching more natural behavior CC license http://www.flickr.com/photos/evilpe acock/6423510787/
- 31. + Use a live recruit test when motivation is key Pretend that you are looking for bookkeeping software for your company…
- 32. + Live recruiting tips: Ethnio Use Ethnio to recruit Read all demographics (unqualified people are listed) Answers may reveal unclear screener questions Pause the recruiting if you leave
- 33. + Live recruiting tips Decide how to handle people who have already started using the site Have sales or relevant phone numbers handy See www.remoteresear.ch for templates
- 34. + Surveys
- 35. + Surveys Questionnaire completed by users to learn about demographics, opinions, etc. Use when you need quantitative data
- 37. + A goal-less survey 1. How did you first become aware of Awesome Accounting? 2. To the best of your knowledge, what does Awesome Accounting do? 3. To the best of your knowledge, what is the price of Awesome Accounting? 4. What software were you using in the running of your business when you first considered Awesome Accounting? 5. What software were you looking to replace when you first considered Awesome Accounting, if any? 6. Why did you first consider Awesome Accounting? 7. Before subscribing to Awesome Accounting, did you do a free trial? 8. Are you currently using Awesome Accounting in running your business? 9. Which of the following parts of Awesome Accounting have you used, or do you intend to use, in running your business? 10. How satisfied are you with the following aspects of Awesome Accounting? 11. Please use the spaces below to comment on each of the aspects of Awesome Accounting. 12. How likely are you to continue to be a Awesome Accounting subscriber 12 months from now?
- 38. Used with CC permission http://www.flickr.com/photos/stovak/
- 39. + Tip: Survey process Ask questions that people can accurately answer What did you eat for breakfast 4 days ago? vs. Do you typically organize your emails into folders? Include an “N/A” option Avoid mandatory questions Ask permission to contact for follow-up questions Pilot test your survey
- 40. + Diary studies
- 41. + Users record activities, problems, or feelings over time CC license http://www.flickr.com/photos/bitchcakes/5457865373/ sizes/o/in/photostream/
- 42. + Diary studies: when to use Seeing experience over time Researching and buying a car Infrequent tasks / non-core tasks To identify places to dig deeper with other studies
- 43. + Diary studies will show you the bigger issues
- 44. + Diary studies: Tips Reserve time to follow up on unclear entries Interact with participants to maintain engagement Continue participants who are useful Ask for screenshots
- 45. + Diary studies: general guidelines Compensation At least $25/week Can increase over time Length: Can keep people engaged for several weeks if not months Recruit: At least 10 More depending on your goals
- 47. + General research tips: planning and logistics Start with goals Pilot Consider recording with permission Have someone else take notes if possible
- 48. + General research tips: facilitating Ask open-ended questions Ask neutral questions Probe for more information Ask for an example Be genuine Show that you are on the user’s side
- 49. + How to get started
- 50. + What is keeping you from getting started?
- 51. + Getting started Start lean and build up Ask forgiveness, not permission from stakeholders Find users who will help for free Get your stakeholders involved Don’t worry about being a research expert
- 52. + More info Books, links, and stuff
- 53. + Quick read to get started UX for Lean Startups: Faster, Smarter User Experience Research and Design by Laura Klein Short. No UX background required.
- 54. + General references on qualitative research Understanding Your Users: A practical guide to user requirements Elizabeth Goodman, Mike Kuniavsky, Andrea Moed by Catherine Courage & Kathy Baxter 700+ pages. Each method = 1 chapter. Observing the User Experience, Second Edition 500+ pages. Each method = 1 chapter
- 55. + On field visits User and task analysis by Hackos and Redish Contextual Design by Beyer and Holtzblatt
- 56. + Live recruit testing and remote research Remote Research: Real Users, Real Time, Real Research by Nate Bolt, Tony Tulathimutte The general reference books I mentioned don’t seem to cover this topic. Look at www.remoteresear.ch for templates
- 57. + On interviewing Steve Portigal’s “Interviewing users”
- 58. + On surveys Coming in 2014: “Surveys that work” by Caroline Jarrett Caroline’s podcast with UIE on surveys
- 59. + Let me know how it goes! anance@gmail.com @amandaux
Editor's Notes
- #5: Across all parts of the lifecycle
- #7: Slides will be available
- #10: SLIDE IS NOT GOSPEL. (Checkmarks are a generalization.)Identify unknown needs = things the USER doesn’t realize they needNatural environment = USER (not necessarily researcher) is in their natural environmentReal motivation = the user needs or wants to do this task; not a task given by researcher.Minor problems apparent -- things like the user’s eyes looking around the screen for a couple seconds before finding a buttonThere IS overlap in these methods. And the data you get is not the only thing to consider; time, resources are also a big driver of which method is used.
- #13: If you skip this step, you could have a product that isn’t useful to people.
- #14: What to build? You could just ask people what their problems are. But people aren’t good at realizing some of their problems.
- #15: Example: We wanted to create a brand new product and didn’t know what kind of product we wanted to build. Interviewed small business owners about all aspects of their business. We asked business owners about their biggest pain points but decided to solve something they didn’t actually mention as a problem. We saw that they were using several different tools to manage their business and entering the same data into more than one application. We decided, what if wehandleall of those tasks in a single application?
- #16: Observation helps because people can’t describe in detail what they do. Observation helps because the researcher doesn’t always know the right questions to ask.In one study, I asked business owners lots of questions about their invoices: when do you send them, how do you handle overdue payments, etc. I even got copies of their invoices. And there were some aspects of bookkeeping that we watched users do. But I never watched a customer SEND an invoice. If I had watched enough customers, I would have seen that they were sometimes emailing an invoice to multiple people at their client’s company. When we designed the application, we assumed that invoices would need just one recipient. WRONG.
- #17: This person was NOT our customer. Do you think she would have showed us her business’s financial data if we had interviewed her on the phone instead of face to face?
- #18: Recruiting: Occasionally, I use an external recruiting company, but I still talk to the participant myself before visiting. Once when I called in advance, the woman who had been recruited as a “small business owner” told me she was not working. If I had not spoken with her personally, we would have visited her home and learned that we couldn’t complete the session with her. Saving work - so you can watch them work.Not cleaning up - seeing all the mess is part of understanding user behavior. Post it notes and cheat sheet may help you understand important pain points.
- #19: Example lets you verify that you understand what participant said. It also keeps them from generalizing, which can gloss over important details.
- #20: Sometimes, field visits can be put you in slightlyawkward situations, especially if meeting in people’s homes. Roll with the punches and always be professional regardless.
- #21: Remember, our users are business owners.
- #24: You make certain assumptions when you design. Users will always use products in ways that you didn’t expect. EXAMPLE: Participant’s invoice list: draft invoices with the SAME client name over and over. We didn’t account for users who do all of their work for the same client.Note that in a survey, she might have asked for a specific field to be added to invoice list, but we wouldn’t know why she needed that.
- #25: Call participant directly if needed
- #26: GoToMeeting - MOV for MacWebEx: Separate meetings = not really separate?WebEx: Recurring meetings must be at a same time daily, weeklyAdobe Connect: Users may want to call in with computer audio
- #31: Explain what an intercept test is. The version of this that involves a facilitator -- ad on the site asking for participants.Call the participant, ask them why they came to the site, ask them to continue and think out loud.
- #32: Even asking users to pretend can give you some good information, if they are target users. E.g., does the information make sense?Live recruit results: Some people just watched the video. Usability test in lab: More people were thorough. Less variation in the amount of time spent on site.
- #33: Story: went to the copier. Gone maybe 4 minutes. Participant was already done looking at the site.
- #34: *** You may get people filling out the form who don’t really want to participate in the study
- #37: Focus = questions must support survey goalsShort = 10 questions or less, depending on how hard it is to answer the questions
- #38: The first question relevant to the survey goals is #10.
- #39: You must focus the survey because ifusers get bored, they will:Get lazy and give you bad information ORStop the survey altogether
- #40: People WILL answer your questions, for better or worseN/A: If it doesn’t apply and you force a choice, your data will be skewedPilot - I have never piloted a survey and found it to be perfect.
- #42: Purpose of one of my diary studies was to learn what problems people were having while first starting to use the software. Had them answer a few basic questions in a Word document each time they encountered an issue. Emailed document to me at the end of each week x up to 8 weeks.
- #44: Sample from an accounting software diary study
- #48: Pilot - not just to test out the test questions and prototype, but also the technology. The one time I didn’t pilot a remote test, first participant couldn’t even access the product on our staging server.
- #49: Despite Dumas and Loring’s advice to respond to participants in a neutral way, I feel that I may start to sound robotic if I’m too strict about this. So, I error on the side of being genuine / human when responding to user suggestions and comments.Show that you are on the user’s side = validate what they say. Tell them that what they do or think makes sense, even if they are being self-deprecating. Validation is important for maintaining rapport and getting the user to talk freely.
- #52: Start lean: 1) takes less time, 2) helps you build credibility and get buy inGet stakeholders involved; if they watch any part of what you’re doing and see value, you may start to get their buy in. You do need some sense of what you are doing. But don’t be afraid that you’re going to ask a question the wrong way or not do everything perfectly. Read what you can, then TRY IT and learn from your mistakes.
- #55: UX for Lean Startups: Faster, Smarter User Experience Research and Design, by Laura Klein