Usability Testing: Backbone of UX Research - DevUp2018

Usability Testing
The Backbone of
UX Research
Danielle Cooley
09 October 2018

Usability Testing: The Backbone of UX Research | 09 October 2018 | @dgcooley | #devup2018
Evaluating a product or
service by testing it with
representative users.
3
Usability
Testing

Participants will
try to complete
tasks while
observers watch,
listen, and take
notes.
Goals:
• identify any usability
problems
• collect qualitative and
quantitative data
• and determine the
participant’s satisfaction
with the product
4

5

6

7

8

9

Confirmation Bias
Tendency of people to seek or
believe information that
confirms their existing beliefs.
10

Acquiescence Bias
Participants agree to whatever the interviewer
presents to them in order to:
• be liked
• pass the “test”
• earn their stipend.
“This was
really easy.”
“I understood
that clearly.”
“10 our of 10!”
“When can I
buy this?!”
11

Central Tendency Bias
People are hesitant to
commit to extreme positions.
12
(Find me later if you want
to talk about all of the other
problems with this survey.)

Social Desirability Bias
People lie when you ask
them about sensitive topics:
• sexual habits
• drug, alcohol, or tobacco
use
13
… and not-so-sensitive
topics:
• internet/phone use
• exercise & eating
habits
• compliance with rules

Hawthorne Effect
When people know they are
being observed, their
behavior changes.
Additional observers
magnify this effect.
14

Evaluator Effect
Different evaluators find
different issues.
15

Selection Bias
Those who
participate in a
research activity are
relevantly different
from those who
choose not to or are
unable to
participate.
16

17

18https://www.userfocus.co.uk/articles/usability_test_plan_dashboard.html

Product
Prototype
Subsection
Specific feature

Justification for
business stakeholders.

21

Determining the
Research Questions
How well or poorly do
people understand
Feature X?
Which elements of the
checkout process are
confusing?
How frustrated do people
feel after interacting with
this product?
Do people understand the
different pricing options?
What questions do users
still have after reading
about the app?

• REPRESENTATIVE users
• Recruit for diversity
• Age
• Gender
• Race
• 5-8 per user type for qualitative
usability testing
• Disability
• Education
• Geography
Use “near
users” only
as a last
resort!

Testing
• Participant hardware
• Participant software
Observation
• Extra monitors
• Screenshare software
• Notetaking tools
Recording
• Cameras
• Microphones
• Screen recording software
• Storage hardware

But also:
• Timing & order
• Stress cases
• Uncertainty
Most Frequent
Most Profitable
Strategically Important
Watch thetask phrasing!

• Recruiter
• Greeter
• Moderator
• Notetaker
• Videographer
• Stakeholder wrangler
• Findings analysts
• Consider stress cases

Testing
Where
- research facility
- conference room
- remote (URL)
- remote observation?
When
- date(s) and times
- specify time zone
Reporting
Where
- conference room
- remote (URL)
- storage of report
When
- date(s) and time(s)

• Greetings
• Refreshments
• Nondisclosure
• Recording consent
• SUS
• UEQ
• NPS
Review issues and
questionnaire
responses
Friendly chatter;
relevant
background
WATCH
and
LISTEN

29
http://bit.ly/EditableUsabilityTestingDashboard

30
Do a
dry run
!!!

31

32
#NotALawyer
Know the law
Geographically and
situationally dependent.
Special
considerations
for children and
some other
populations.

33
Informed Consent
Participants get all
information in a form
they can understand.
• Purpose of research
• Right to decline/withdraw
• Risks/discomfort to them
• Benefits to participant
• Limits of confidentiality
• Incentives
• Contact information for
follow-up questions

34
Confidentiality and
privacy
• Preserve anonymity
• Never use surnames
(in session or report)
• Blur faces if necessary
• Honor agreements re: video
and data distribution
• Take practical security
measures with recruiting
and payment information
and stored recordings

35
Participant comfort
Physical and
emotional

36

37
If you’re
talking
you can’t
be listening

38

39

40
Confirmation Bias
Tendency of people to seek or
believe information that
confirms their existing beliefs.
Recruiting
• choosing highly
experienced participants
Tasks
• easy tasks
• no stress cases
• leading phrasing
Observation
• selective listening
• downplaying or dismissing
issues

41
Acquiescence Bias
Participants agree to whatever the interviewer
presents to them in order to:
• be liked
• pass the “test”
• earn their stipend
Self-blame
High rating scores
You can pay
them first!

42
Central Tendency Bias
People are hesitant to
commit to extreme positions.
1 2 3 4 5
1 2 3 4 5 6 7
rating scales
comments

43
Social Desirability Bias
People lie when you ask
them about sensitive topics:
• sexual habits
• drug, alcohol, or tobacco
use

44
Hawthorne Effect
When people know they are
being observed, their
behavior changes.
• Hide observers in a separate room
• Downplay their numbers
• but still must disclose

Evaluator Effect
Different evaluators find
different issues.
45
Use multiple evaluators.

Selection Bias
Those who
participate in a
research activity are
relevantly different
from those who
choose not to or are
unable to
participate.
46

47
danielle@dgcooley.com dgcooley.com

Workshops
48
Research Design Evaluation

Usability Testing: Backbone of UX Research - DevUp2018

More Related Content

Similar to Usability Testing: Backbone of UX Research - DevUp2018 (20)

More from Danielle Cooley (20)

Recently uploaded (20)

Usability Testing: Backbone of UX Research - DevUp2018