Moderated user testing: do's and don'ts
4 likes1,768 views
The document provides a comprehensive guide on moderated user testing, covering basic concepts, types of tests, the importance of user testing, user recruitment strategies, scenario writing, and moderation techniques for successful tests. It emphasizes the need to engage with the right participants, avoid common pitfalls, and adapt scenarios to user input. Additionally, it highlights the value of user testing in improving websites and apps and offers resources for further learning.
Moderated user testing: do's and don'ts
- 1. Do's & don'ts of moderated user testing Karl Gilis @agconsult
- 3. > 3.500 user tests Since 2001
- 4. Some of our clients
- 5. What will we be talking about? 1. Moderated user testing: the basics 2. What can you test? And where? 3. 8 reasons why user testing is so valuable 4. Recruiting the right users & making sure they show up 5. Writing the scenario 6. Your job on the test day itself 7. What to do next?
- 6. 1. Moderated user testing: the basics
- 7. 3 key methods to make your website better AB-tests User testing Expert review
- 8. What are moderated user tests? • You arrange for a number of people to perform a series of typical tasks on your website, intranet, app, … • The test takes place on an individual basis. • A moderator asks the questions and leads the user test. • Observing the user while he's using the website is key to this method.
- 9. 2. What can you test? And where?
- 10. What to test? Your website, intranet, app, … Those of your competitors Early prototype to final design
- 11. On every screen!
- 12. In-person user testing Guerilla testing Remote user testing 3 types of moderated user testing
- 13. User testing myth You need a test lab
- 14. User testing myth #2: Don’t test in the client’s office
- 15. 3. 8 reasons why user testing is so valuable
- 16. 1. Experts don't know everything
- 18. "The more I learn, the more I realize how much I don't know." Albert Einstein
- 19. 2. Launch something that people will like to use
- 20. Old website New website 8% less sales Investment: 150 million GBP
- 22. 3. Discover problems that really matter (but are hard to find with other methods)
- 23. 4. Discover why something is a problem
- 24. 5. Generate ideas for AB-testing
- 25. 6. Facts trump opinions
- 26. 7. Ignoring your own clients is difficult
- 27. 8. Convincing and durable proof
- 28. 4. Recruiting the right users & making sure they show up
- 29. Don't test with your mother!
- 30. Profile of the test users • Most important: • Belongs to your target audience • Mix between customer/non-customer • You also want a good mix between • Gender • Age • Educational level
- 31. Relax. Don't become too fixated on a specific mix of test users or on a ridiculously precise profile.
- 33. Speak to them beforehand!
- 34. Avoid people who are in it for the money
- 35. Don't use the word 'test'
- 36. Other recruiting pitfalls • Avoid professional testers • Don't recruit people with an alcohol problem • Call every participant the day before the test to reconfirm • Make sure you have a back-up participant
- 37. How many people do you need?
- 39. Iterate
- 40. How do you find those users? • Pop-up or slide-in on your site • Mail your newsletter subscribers • Mail your client database • Facebook, Twitter, LinkedIn, … • Your call centre • Notice board of your physical location • Recruitment agency
- 41. Let's talk about money • Cash • Gift vouchers • Coupon codes • Your own product, gadgets …
- 42. 5. Writing the scenario
- 43. 1. Set the scope of the test
- 44. 2. Focus on the important things
- 45. 3. Make a list of possible questions
- 46. Typical pitfalls when making a scenario • Using words that are on your website • Tasks that are too specific or aren’t relevant for the participant • Book a citytrip to Rome for Valentine for 2 people, half board? • Buy a pair of running shoes, size 38, in pink • Not using plain language • Too many tasks that are too difficult
- 48. 6. On the test day itself
- 49. Relax. You are prepared.
- 50. As a moderator you have to combine 4 'personalities'
- 51. 1. Steward
- 52. 2. Switzerland
- 53. 3. Scientist
- 54. 4. Reporter
- 55. Steward: make the participant feel at ease • Ask if it was easy to find the location, offer them a refreshment… • Explain what will happen: research of an existing site, app, wireframes (I call it a 'draft version' of a website), etc. • Never use the word test. It's not a test, you're doing research. • Make it clear you're researching the website or app, not the user. They can't do anything wrong. There are no wrong answers. • Assure the user you were not involved in building the site (even if you were). Nothing the test user says will therefore flatter or insult you.
- 56. Steward: start gently • Move from the introduction to an interview • Ask the test user a number of questions on his experience with the site, on similar sites, on surfing in general. • Refer to what you already know about him/her (during the recruitment phase) • Start the test with a familiar or easy task • If the user is familiar with the site or app (or related ones) • Do you remember why you used the website on the most recent occasion? • Did it work well then? • We will try and do the same now. • If not, start with an easy task
- 57. How to deal with the scenario? • The scenario is a guideline, not a straitjacket • Try to adapt your tasks/questions to the user • Leave room for the test user's input • Ask a simple task after a failure • Mix up the order of your questions
- 58. Your job once you've given a task? • The best thing you can do is to just shut up. • Give people the time to think and to express their thoughts • Do not interrupt people, even if they’re on the wrong track • Of course you can ask additional questions • When depends on the situation • Interact based on their behavior, facial expressions, what they say, how they say something, …
- 59. Of course there's so much more… • How to respond to typical situations? • How to deal with 'difficult' users? • In what situations can you help the user? • How useful is it to ask questions after each task? • How to make sure the observers understand what's happening? • How to take notes? • How to make sure your observations and conclusions aren't biased? • How to report on a user test? • How to find solutions for the problems you discovered?
- 60. What to do next? • Learn more during our deep dive workshop • London – October 11 2016 • Berlin – November 7 2016 • Mail karl@agconsult.com if you want an in-house training • Do a user test! • Never forget to AB-test the ideas you have to tackle the problems you discovered.
- 61. Get my free ebook now 10 conversion tips that work on every website 15 cases from the real world 6 best practices from leading websites www.agconsult.com/ebook.html www.slideshare.net/agconsult Already more than 3.216 downloads
Editor's Notes
- #8: Terminology E.g.: the heading on a button Position of an interface element E.g.: picture of a product on the left or right side of a screen To add or omit particular information E.g.: microcopy on a form, USP in an ordering procedure
- #13: The biggest disadvantage of the 'classic' in person user test is that your test participants either have to come to you or you have to go to them. And time is simply money. Remote moderated user testing is an excellent alternative for in-person user testing.Tweet this quote With remote user tests, the moderator and test participant can be wherever they want to be, as look as they're looking at the same website on some screen or other. With screen sharing software that’s a piece of cake nowadays. If your user has a webcam, you can also see their face. Which makes it a good alternative for in-person moderated user testing. Test participants live in different locations / time zones Difficult profile test participants Time is precious
- #14: Most people think you need a test lab with a two-way mirror to do proper user testing. I say: bollocks. That mirror’s not fooling anyone. User testing is an artificial situation, there’s no way around that. But let’s not make that any worse with a two-way mirror. - You need a user test lab No, you don’t. Actually: please do not test in a lab. User testing is an artificial situation, there’s no way around that. But let’s not make that any worse.
- #15: Testing in the client’s office intimidates the test participants I won’t say someone very timid couldn’t be intimidated by a big brabd’s posh office. (I would argue these people won’t really volunteer to be a participnat in a test very often either.) What I will say is that I’ve heard test participants say some really horrible things about a website in really lovely looking offices.
- #17: Do we really need to test it? I'm an expert! Especially when you’ve been an expert for quite some time.
- #18: You start to think all you as an expert have to do is to look into your crystal ball of infinite knowledge and the answers will just pop out. But that’s not really true. The only crystal ball there is is data – user research data.
- #19: No matter how much expert knowledge and experience you have, you can’t know everything. You can’t always judge situations perfectly. That’s a lesson I've learnt in more than 15 years as a usability expert. You can read more on that topic in our article 'Experts don't know everything, not even usability experts'.
- #20: Changing without testing = gambling?
- #22: Don’t be afraid to reach out to your users.
- #23: Structure-can people find what they're looking for – understand navigation (content and how it works) – understanding copy – what does the company try to sell – why would you buy it – forms- …
- #26: Clients can always dismiss expert advice with the argument "Yeah sure, but that’s just your personal opinion, right?". User tests reveal facts. And facts trump opinions any day of the week.
- #27: Or at least, ignoring your own clients is more difficult to do than ignoring expert advice. If you have sees with your own eyes that 5 out of 6 people have trouble using the filter navigation in your online shop, there's no way you can't realize this is a problem you have to fix.
- #28: A video that shows your customers and potential customers having trouble with your website, cursing your interface? That has immense power. That kind of video will shut up even the most die-hard, ego-tripping know-it-all experts and directors during board meetings. Also, you can re-use the recordings at any stage of the project. Whenever that same old discussion rears it's ugly head again, you just whip out that recording: “Remember, this is what we saw during user testing...”. Keep hammering away on that nail.
- #31: Must obviously be supplemented with additional criteria depending on the site or app to be tested Experience with surfing on a table or with a specific app – shops online or not / watch tv on iphone
- #33: How can you avoid ‘bad' test users? Google all candidates and check their LinkedIn profile IT, web or communication professionals They tend to behave like experts rather than users in the course of the test. Competitors Employees from competitors on a 'spying mission' are never part of the plan. Don't recruit people with an alcohol problem
- #34: People who are difficult to understand The whole point of a user test is to gather feedback. If you cannot understand the person, you won't know what they are trying to say. People that are too shy. Call all candidates to confirm their participation. Then you will hear their voice and have an opportunity to assess what they are like. Tip: Call them under the pretence that you want to check whether they can still take part and that you will confirm their participation at a later date.
- #38: Testing 5 users gives insight into 85% of the usability problems.
- #39: It is best to test at least 5 subjects. This prevents you from drawing hasty conclusions. The maximum number of test subjects depends on the complexity of the site or application to be tested and the diversity of your target audience. E.g.: job seekers versus companies, private individuals versus professionals, etc. Iterate:
- #40: It's better to do 1 user test every week and tackle the problems you've found in between, than 3 consecutive days of testing on the same version.
- #42: It is standard to pay 40 to 50 euros per person for a user test lasting between one hour and one hour and a half.
- #44: what do you want to find out? - What are the new elements on the site that you want to test? - What are you unsure about? – What are the problem areas of your website, based on GA, user session recordings, heatmaps, …
- #45: Focus on key tasks What do most people visit the site for? What is researched most frequently? What is important for your business (will result in more income, or reduce costs) The main focus must be on improving those things. Do not ask questions about exceptions. It is not a problem if things that people rarely need or that only a minute group of users is looking for are harder to find. It is impossible to make everything equally easy to find on a website.
- #47: In no case should the questions be read out during the test. Practice the questions you will ask and the precise words you will use. The tasks must sound very natural to the test user. ‘It has been ages since you and your partner went away for the weekend. Do you want to book a weekend trip for the two of you?'
- #48: Do a dry run. Rehearse your question. Ask them over and over again. As natural as possible. Learn your introduction by heart. Practice with real people.
- #51: It's better to do 1 user test every week and tackle the problems you've found in between, than 3 consecutive days of testing on the same version.
- #52: Ensure that the test user is feeling at ease, in particular: be patient...
- #53: Be as neutral as possible; do not ask any suggestive questions. Do not express a preference.
- #54: Collect data.
- #55: Narrate and explain what is happening for the people in the monitoring area.