Why you should always test with users...

Editor's Notes

• #3: Pet hate - “User testing” GDS = Government Digital Service in the Cabinet Office
• #4: I sometimes like to reframe it as Testing with Customers (not “users”) Makes it commercial Customers = revenue
• #6: 3 of my favourite things….
• #10: Senior UX Consultant at amaze Used to be a front end dev Drifted into UX in 2005 Been doing this for 10 years Joined amaze in April 2012 Amaze founded in 1995 Lots and lots of people (about 220!) Offices in Manchester, Liverpool & London This is the Manchester office opposite the old Hacienda Full service consultancy for B2C and B2B: Strategy & creative, Design & build, Digital commerce and technology, Integrated comms, Global implementation support We’re quite good…
• #11: One of the NUX organisers Free UX meet-ups and big all-day conferences in Manchester, Leeds and Newcastle
• #13: Most common form of usability testing 1 hour sessions Lots of findings Lots of dedicated consultancies for this Also many digital agencies do it (we do it) Number of one hour session, with full written reports Audio and video capture, edited highlights etc etc Often clients watch from behind one way mirror Worked with BBC for Cbeebies William Hill anecdote of screaming at screen – needs to be soundproofed
• #14: This sort of thing Pros: Users are often in a more informal environment (comfortable and relaxed) Great for bringing in teams together to observe users (and discuss) Help and support when defining tasks and scenarios Less likely to introduce moderator bias Detailed external provider analysis and reports More likely to use advanced data collection (Eye tracking) Cons: Can be costly Less likely to want to invest in early stage testing (Prototypes and early beta code) Less likely to perform iterative tests
• #15: Browser toolbar & plugin gives users tasks and captures screen & video/audio Consultancies have a pool of people they use Good for quantitative testing – large generate enormous amount of data and feedback from participants Answering specific questions about how people use a user interface for relatively simple and straightforward tasks
• #16: “sciency” Clients love it
• #17: Perception vs reality
• #19: Despite the complicated stuff, we know it should be done But how can we make it manageable?
• #20: What Users Do & Econsultancy Report into attitudes and approaches to UX
• #23: One area was why don’t you test? More reasons: Findings not actionable Will create more work “It can’t be that bad” “You should already know”
• #24: One area was why don’t you test? More reasons: Findings not actionable Will create more work “It can’t be that bad” “You should already know”
• #25: Our answer to this is 15 15 Environments – has ecological validity, behaviours that occur in natural settings, not lab environment We managed to achieve some of these, but not all
• #26: Sweetspot 5 users = 85% of issues / Multiple iterations Saturation at about 10 users Exceptions: Quantitative studies (aiming at statistics, not insights): Test at least 20 users to get statistically significant numbers; tight confidence intervals require even more users Card sorting: Test at least 15 users Eyetracking: Test 39 users if you want stable heatmaps Any users are better than none
• #27: The solution
• #28: Take ASICS from non-transactional to transactional Requires whole sales, warehouse and distribution network, and customer services establishing America - live Japan - live Europe - soon Australia Then on to China, South America and rest of world
• #29: Put these up in the lifts and the lobby
• #30: Use these Only need Vimeo if you’re sharing the videos Only need the Pro version of Vimeo if you want to password-protect them
• #31: We did some stuff – it looked like this
• #32: Identified 2-3 key user journeys & features Prototype new features for journeys – medium fidelity Recruit from same office building – posters in lobby Test in spare meeting room in Amaze office Slots 12 - 2pm & 4 - 6pm Brief, focussed report - bullet point list will suffice Export & upload videos & prototype
• #33: Identified 2-3 key user journeys & features Prototype new features for journeys – medium fidelity Recruit from same office building – posters in lobby Test in spare meeting room in Amaze office Slots 12 - 2pm & 4 - 6pm Brief, focussed report - bullet point list will suffice Export & upload videos & prototype
• #34: Identified 2-3 key user journeys & features Prototype new features for journeys – medium fidelity Recruit from same office building – posters in lobby Test in spare meeting room in Amaze office Slots 12 - 2pm & 4 - 6pm Brief, focussed report - bullet point list will suffice Export & upload videos & prototype
• #37: Eg: IT and design bods Unless they really are your audience
• #39: Eg: IT and design bods Unless they really are your audience
• #40: 15 minutes isn’t long - 2 mins will be intro - People like to talk Focus on 2 or 3 key areas for iteration You will have more iterations to test other stuff Small guerrilla throughout the project to keep it on track, but then a big lab test at the end to rubber stamp it for key stakeholders
• #41: 15 minutes isn’t long - 2 mins will be intro - People like to talk Focus on 2 or 3 key areas for iteration You will have more iterations to test other stuff Small guerrilla throughout the project to keep it on track, but then a big lab test at the end to rubber stamp it for key stakeholders
• #43: Basic principle of testing “You need a new pair of running shoes, what would you do?” not “Click on ‘running’ and find a pair of green size 8s suitable for road or trail”
• #44: ABC BAC CAB Etc Eg: find returns info – could have seen during “buy a shoe” task Order effects
• #45: Important thing is prototype should look realistic enough to be usable Don’t test wireframes Having said that, you can “explain away” inconsistencies or quirks during the session
• #46: You’re looking for something you can make a decision based on Not statistically valid rigorous data Consider remote testing if you need to thrown numbers at something
• #47: Never cease to be amazed by what people say Have open mind Be flexible Its fun, amusing, and exhausting