Usability testing / Nearly everything you need to know to get started

Usability testing
Nearly everything you need to know to get started
Rebecca Destello

Rebecca Destello
•  Restaurant manager
•  Front-end web developer
•  User researcher
•  Aﬃliate lecturer, UW HCDE
•  Chair of Puget Sound PSSIGCHI
•  Manager, Research & Insights,
Atlas Informatics
R. Destello | Usability Testing

Atlas Recall is a searchable photographic memory
for your digital life — if you see it, you can search
it and share with anyone, from anywhere.
atlas.co

Are you experienced?

Tonight’s lecture
What you’ll need to know
to do this as a professional

Why do we do
usability testing?

Because there is a science to how people use things.

usability testing:
the ﬂow
Identify
problems in the
system
Create a study
Recruit and
schedule target
users
Test with
participants
Analyze +
Synthesize
Report out
Evangelize
ﬁndings!
Repeat

The one book you need:
Handbook
of Usability Testing
by Jeﬀery Rubin and Dana Chisnell

Planning

•  Identify key stakeholders
•  Scope and priorities
•  Identify & recruit target users
•  Establish a timeline
•  Identify usability concerns
planning a usability study

It is critical to get key players involved up-front
•  Project sponsors
•  Product owners
•  People who will have to act on the ﬁndings
identify key stakeholders

How do we
generate support
for usability testing?

1. Start running studies!
2. Get people to observations
3. Always be prepared with talking points; include
ﬁndings from other studies that made a critical
diﬀerence
4. Be a good team player - be willing to compromise
and negotiate
5. Map usability planning to the company and product(s)
5 steps to getting buy-in

Think about:
•  What is the political landscape? How much buy-in do
you have from the stakeholders?
•  Has usability been done before?
•  What questions are you trying to answer?
•  What is your budget?
•  What time constraints are you working with?
scope and priorities

Tips for identifying areas of concern:
•  Ask your stakeholders
•  What do you want to learn from this test?
•  What are your greatest fears about this product?
•  Are there diﬀerent points of view in the group about this
issue?
•  What are you willing to change as a result of this test?
•  Mine customer feedback
•  Perform a task analysis
•  Evaluate the product (heuristic evaluation or cog. walkthrough)
identifying usability concerns

!

Bad recruiting is bad testing

•  Find your target user by asking your client to add
a survey to their website
•  Use Craigslist (with caution)
•  Ping your social networks
•  Go where your users might congregate
And, write a good screener…
tips to ﬁnd your target user

The screening questionnaire needs to be
•  Specific, inclusive of all necessary
criteria
•  Contain boundary conditions
•  Contain exclusion criteria
May need to stand alone
•  May be passed off to others, firm,
colleague
Can be lengthy
•  10 pages
•  28 questions
screeners

Tips for writing a good screener
•  Ask the elimination questions ﬁrst
•  Eliminate conﬂicts of interest
•  Screen for experience
•  Recruit based on behavior and attitudes
•  Screen out “professional” users
•  Write a good screener (bit.ly/uxmattersrecruiting)
screeners

usability test participant incentives
Why should you give an incentive?
•  Participants are helping you by giving up their time
•  Participants will leave with a positive attitude
•  Participants will tell their friends (larger pool of
participants for future studies)
•  Best way to get qualiﬁed, representative participants

usability test participant incentives
What should the incentive be?
•  Incentives are determined based on the client /
budget
•  Example:
–  Microsoft gives out free software
–  Boeing gives out movie tickets
–  Chocolate (at the very least)
How much should an incentive be?
•  Based on budget and target audience

Your timeline will be determined by
•  The complexity of the product
•  The broadness of the scope of the test
•  What set up will be needed (facilities, travel, special equip, etc.)
•  Level of diﬃculty in recruiting the target users
•  # of participants needed
•  How formal the report out will need to be
•  Related deadlines (design, dev, and even holidays!)
establish a timeline

Conducting

what’s in the usability study?
ü  Product or assets to test
ü  Participants (5 works if doing iterative testing, if only 1 user group)
ü  Moderator script with scenarios and tasks
ü  Pre/Post task questionnaires
ü  Checklists for study setup
ü  Participant documents (Informed Consent, NDAs, Questionnaires)
ü  Data collection tools
ü  Participant incentives

•  Greet and make participant comfortable
•  Explain study, setup, and complete forms
•  Explain Thinking Aloud Protocol
•  Remind them they are not being tested
•  Begin tasks
•  Wrap-up
•  Questionnaire, if applicable
•  Final Comments/Questions
•  Thank them profusely
•  Provide gratuity
•  Parking validation, if applicable
testing framework

what about scheduling?
Allow time during the study:
•  Time for test (30 minutes - 1 hour)
•  Time in between (1 hour between participants)
–  Setup for the next study, check in with observers, clean up
notes, start examining data while it’s fresh
E X A M P L E

stay organized with a checklist
Why use a checklist?
•  Checklists are good, reusable tools for any usability study
•  Checklists outline all things that must be completed before,
during, and after study
•  Helps ensure study consistency and increases internal validity of
the study from day 1!
Key Takeaway: Even if you don’t think you’ll need it, create a checklist of
all the things you need to do for study (you’ll be surprised what you forget!)

participant documents
Before Testing
•  Pre-Test Questionnaire (optional)
•  Informed Consent/NDA (required)
During Testing
•  Post-task Questionnaires
After Testing
•  Post-Test Questionnaire

What’s the role of
the moderator?

so what does the “moderator” do?
ü  Interacts with the participant
ü  Ensures the participant’s comfort
ü  Sets the pace and tone of the study
ü  Administers scenarios, tasks, and questionnaires
ü  Handles prompting and working with the participant to
understand the reasons behind problems.
ü  Provides clariﬁcations so note taker can capture data
ü  Takes care of the participant!

why is moderating challenging?
•  Goal is unbiased feedback
•  Participants need to be comfortable, but the
test situation is often unsettling
•  You need to know what the participant is
thinking, but you can’t read their minds

intervention: what are they doing?
When and how to intervene during a test:
•  In general, it’s preferable to let participants decide
how much time to work on a task
•  Some exceptions
–  If they are getting very frustrated
–  If you feel that you’ve got data on the task at hand and you
want to keep going and get to other tasks
–  If they’ve wandered so far down a bunny hole
they are unable to proceed with the remaining tasks

prompting: guidelines
When prompting, you want to…
•  Get the user to elaborate on something that is unclear
Without…
•  Introducing bias
•  Inﬂuencing the user’s behavior
•  Taking over as the expert or primary speaker
•  Unnerving the user through implication that she
did something wrong

prompting: techniques
Provide Acknowledgment Tokens at natural intervals
•  Mm hmm, I see…..
Focus on Questions
•  How do you feel about that?
•  Tell me what you are thinking…
•  Tell me more about that…
Repeat their word or phrase back to them as a question
•  They say: That was confusing / You say: “confusing…?”
Make the alternative explicit
•  Was this task easy or diﬃcult?

Keep in mind…
•  People cannot reliably reﬂect on their own cognition
•  Without video to playback questions like “what were you
thinking when…” or “why do you think you…” you will ﬁnd
you often have missing / bad data
•  Participants will struggle to come up with a believable
rationale while being unaware of the true cause for their
actions
prompting: words of warning

participant questions
Participants sometimes ask the moderator for help. Turn their
question into a question.
How would you respond to the following?
•  Is that right? Did I do it right?
•  Am I supposed to keep going?
•  What should I do now?
•  Should I click here?
•  What is this?

The most important thing to remember
about moderating is this:
Being in a study is stressful for the participant – their
comfort level and state of mind are vital. When people
are stressed, their behavior/memory are impaired.

Things to keep in mind while moderating:
•  Be patient
•  Be excited about the process of discovery
•  Be “small” in relationship with the participant
•  Be as neutral as possible
•  Be professional
•  Be caring to the participant’s needs
Important: The newer you are, the less you should talk.
moderate with care

q  Does the moderator use relaxed body language?
q  Does the moderator encourages/discourages responses?
q  Does the moderator follow-up (when allowed) and dig into ambiguous
statements or promising topics or ideas
q  Does the moderator have any verbal or behavioral tics that would be
distracting?
q  Does the moderator provide excessive reinforcements?
q  Is the moderator aware of expressions of support, sympathy, joy, anger, and
other emotions?
q  Does the moderator recover from miss-steps well.
q  Does the moderator make the interviewee comfortable during the initial
brieﬁng and questions.
q  Does the moderator clearly explain the limits of his/her ability to help the
participant?
q  Does the moderator follow protocols for prompting?
q  Was the moderator sensitive to cultural/org. issues?
CHECKYOURSELF
beforeyouwreckyourself
R. Destello | Usability Testing – Moderator Checklist credit: Chauncey Wilson

moderator script: the essentials
Prepares participants for the session
•  Describes what to expect (how long, what they’ll be doing, if they’ll be observed, if
they’ll be videotaped)
•  Gives think-aloud instructions
•  Lets them know their rights
Why is the script important?
•  Ensures that the facilitator gives the same information to all the participants
•  Stresses that the study is a test of the product, not of the participant

moderator script: the essentials
How to write unbiased, non-leading usability tasks (at the
10,000 ft. level):
First think: What is the user naturally trying to do at this step? Then –
•  Make them short (hard to remember long tasks)
•  Write tasks in the user’s natural language
•  Avoid jargon
•  Don’t use words found in the system

For a successful study…
•  Make sure to know what data you are looking for (what questions are
you trying to answer?)
•  Deﬁne success criteria for each task
•  Pilot the study at least once
•  Do a quick “clean up” of notes after each session
•  Have a plan for eﬀective note-taking so you can spend less time
analyzing data
other considerations

Analyzing + Synthesizing

Data you can capture during the study:
•  Pathways
•  Verbal commentary
•  Think Aloud Protocol (TAP)
•  Time on task (only when not using TAP)
•  Task completion
•  And more
data collection: available data
Data can be gathered using:
•  Handwriting
•  Document
•  Spreadsheet
•  Usability software

data collection: DOs
Do capture:
•  Observations; what you see the participant do
•  Quotes; what you hear the participant say
•  Details; what speciﬁcally caused issues or
prevented task completion

data collection: DON’Ts
Don’t capture:
•  Why you think they are doing what they are doing (if
you aren’t sure, ask the participant)
•  Broad generalizations that don’t contain a
description - “She had a lot of trouble ﬁnding the link”
won’t be helpful when you’re analyzing ﬁndings

Use codes to note
instances of issues
that you’re tracking
Example:
F = failure
E = error message
H = goes to help
Keep an issues
matrix and add to it
after each
participant.
data collection: note-taking tools

data collection: note-taking tools

usability software: Morae

usability software: Silverback

Reporting + Evangelizing

•  Word of mouth and email
•  Executive summary
•  Formal report
•  Presentation
How you choose to communicate results can depend upon
your audience, the culture, timeline, and the potential
“Freak Out Factor”
communicating results

•  Know your user (reader) – what’s important to them?
•  Accentuate the positive, but deliver the news
•  Report the facts (don’t overreach)
•  Be thoughtful about word choice
(opportunity vs. issue)
•  When in doubt, recommend further testing
•  Check grammar and spelling
communicating results: tips

In groups of 2, using a phone 1 person act as the participant, the other as the
note taker and go to Nordstrom.com and complete the following tasks:
1.  You’d like to see the hot new items for this season. Where on the site
would you browse what’s trending this fall for yourself?
2.  You want to buy a new shirt for your 5 year old niece. Where would you
ﬁnd a shirt that might work for her?
Focus on problems in navigation and ﬁlter use.
in-class exercise: Navigation

Don’t forget!
•  Take notes (Where did the participant have trouble?
What did they say? What worked?)
•  Use prompts to keep them talking: “What are you
thinking?” “Say more.” “Is that what you expected?”
•  Don’t say much and listen closely

In groups of 2, using a phone 1 person act as the participant, the other as the
note taker and go to Nordstrom.com and complete the following tasks:
1.  You’d like to see the hot new items for this season. Where on the site
would you browse what’s trending this fall for yourself?
SWITCH ROLES
2.  You want to buy a new shirt for your 5 year old niece. Where would you
ﬁnd a shirt that might work for her?

What was the experience like?

Questions?
RebeccaDestello@gmail.com

Usability testing / Nearly everything you need to know to get started

More Related Content

What's hot (20)

Similar to Usability testing / Nearly everything you need to know to get started (20)

Recently uploaded (20)

Usability testing / Nearly everything you need to know to get started