Journey from subjective to objective: Capturing user experience

Journey

from
Subjective
to
Objective:

Capturing
User
Experience
Digital
User
Experience
(DUX)
Working
Group
in
Collaboration
with
Sciences
Librarian
Indiana
University
– Purdue
University
Indianapolis
Yoo
Young
Lee

Eric
Snajdr
Lisa
Calvert
Andy
Smith
@DUX_YooYoungImage
Credit:
http://www.ploink-‐brothers.com/2016/03/23/get-‐results-‐by-‐not-‐making-‐these-‐web-‐design-‐mistakes/

Table
of
Contents
@DUX_YooYoung
1. Introduction
2. Usability
vs.
User
Experience
3. UX
Method
&
Data
Analysis
4. Results
5. Discussion
&
Next
Steps
6. Q&A

Introduction
@DUX_YooYoung
Background
Information
2014
• DUX
working

group
formed
• Project
of
the

redesign
of
UL

website
initiated
• Surveys
• Google

Analytics
• User
behavior
• Wireframes
• Mockups
• Prototypes
2015.07
Launch
Project
launch User
research Design
&
Implementation Evaluation
• UX
in
the
library

instruction
• Informal
UX
Testing

Introduction
@DUX_YooYoung
Research
Questions?
1. Measure
overall
UX
evaluation
2. Measure
overall
task
performance
3. Difference
between
what
they
said
and
what
they
did?
4. Library
instruction
affects
UX?
If
so,
what
ways?

Usability
vs.
UX
@DUX_YooYoungImage
Credit:
Facebook
– Aménagement Paysager Jarda

Usability
vs.
UX
@DUX_YooYoung
Web

UX
Information

architecture
Interaction

design
Interface

design
Usability
User
experience
for
the
web
is
all
about
how

users
feel
when
interacting
with
a
website
or

interface. (Schmidt,
2012)

UX
Method
@DUX_YooYoung
How
to
capture
UX?
• In
the
library
instruction:
9
different
sessions
• Total
participants:
213
students
• Valid
data:
160
students
Why
in
the
library
instruction?
Valid
data
• Over
18
years
old
• Completed
all
four:
tasks/
pre&post
surveys/
instruction
Participants
• Enrolled
in
sciences
program
• Lower
level:
107
students
• Upper
level:
53
students

UX
Method
@DUX_YooYoung
How
to
capture
UX?
Tasks Pre-‐Survey
Library

instruction
Post-‐
Survey
• Brief
overview
• 6
tasks
under
5

minutes
• Tool:
Screencast-‐
O-‐Matic/
Survey

Monkey
• 9
questions
• 5
minutes
• Tool:
Survey

Monkey
• Information

literacy
• 9
questions
• 5
minutes
• Tool:
Survey

Monkey

UX
Method
@DUX_YooYoung
Tools
used
• Online
survey:
Survey
Monkey
• Screen
capture:
Screencast-‐O-‐Matic
• Analysis:
R
for
data
analysis
and
R
and
Excel

for
display

UX
Method
@DUX_YooYoung
Data
Coding

UX
Method
@DUX_YooYoung
Data
Coding
• Task:
Yes/
No/
Skip/
Non
complete/
Title
of
the
page
• Behavior
• Time

Results
@DUX_YooYoung
Overall
UX
Evaluation
on
Website
(Pre-‐survey
right
after
tasks)
Data
sets:
All
0.00
1.00
2.00
3.00
4.00
5.00
Easy
to
use
Fun
to
use
Helpful
User-‐friendly
Confident
Satisfied
Positive
Aspects
0.00
1.00
2.00
3.00
4.00
5.00
Difficult
to
use
Frustrating
to
use
Useless
Confusing
Frustrated
Uncertain
Negative
Aspects
5:
Strongly
agree
4:
Agree
3:
Neither
agree
nor
disagree
2:
Disagree
1:
Strongly
disagree
0:
N/A

Results
@DUX_YooYoung
Overall
UX
Evaluation
on
Website
(Pre-‐survey
right
after
tasks)
0
1
2
3
4
5
Easy
to
use
Fun
to
use
Helpful
User-‐friendly
Confident
Satisfied
Positive
Aspects
All Lower Upper
0.00
1.00
2.00
3.00
4.00
5.00
Difficult
to
use
Frustrating
to
use
Useless
Confusing
Frustrated
Uncertain
Negative
Aspects
All Lower Upper
α =
0.05
Data
sets:
All
5:
Strongly
agree
4:
Agree
3:
Neither
agree
nor
disagree
2:
Disagree
1:
Strongly
disagree
0:
N/A

Results
@DUX_YooYoung
Overall
Task
Performance
T1:
Find
information
on
how
many
books
you
can
borrow.
T2:
Find
your
subject
guide.
T3:
Find
information
about
the
3D
Printing
Studio.
T4:
Where
would
you
go
when
you
don’t
know
where
to
start
on
your
research
project?
T5:
Find
one
scholarly
article
in
your
major.
T6:
Find
one
database
in
your
major.

Results
@DUX_YooYoung
Overall
Task
Performance
(Right
Answer)
Data
sets:
All
24
26
48
30
26
6
0
10
20
30
40
50
60
0 1 2 3 4 5
5:
Answer
right
5
out
of
5
4:
Answer
right
4
out
of
5
3:
Answer
right
3
out
of
5
2:
Answer
right
2
out
of
5
1:
Answer
right
1
out
of
5
0:
No
right
answer
Scores
#
of
students

Results
@DUX_YooYoung
Overall
Task
Performance
Upper
Level Lower
Level
α =
0.05
Mean
of
Upper:
2.77
Mean
of
Lower:
1.86
P-‐value:
0.0001851

Results
@DUX_YooYoung
Overall
Task
Performance
(Previous
Library
Instruction)
α =
0.05
A B
Mean 2.07 2.12
Variance 1.48 2.01
Observations 46.00 42.00
Pooled
Variance 1.73
Hypothesized
Mean
Difference 0.00
df 86.00
t
Stat -‐0.19
P(T<=t)
one-‐tail 0.42
t
Critical
one-‐tail 1.66
P(T<=t)
two-‐tail 0.85
t
Critical
two-‐tail 1.99
A:
Students
who
have
never
had
library

instruction
before
B:
Students
who
have
had
1-‐3
library

instruction
sessions
before
5:
Answer
right
5
out
of
5
4:
Answer
right
4
out
of
5
3:
Answer
right
3
out
of
5
2:
Answer
right
2
out
of
5
1:
Answer
right
1
out
of
5
0:
No
right
answer

Results
@DUX_YooYoung
A:
Students
who
have
never
had
library

instruction
before
B:
Students
who
have
had
1-‐3
library

instruction
sessions
before
0
1
2
3
4
5
Pleasant
to
complete
Not
pleasant
to
complete
Easy
to
complete
Difficult
to
complete
A B
α =
0.05
Overall
Task
Performance
(Previous
Library
Instruction)
5:
Strongly
agree
4:
Agree
3:
Neither
agree
nor
disagree
2:
Disagree
1:
Strongly
disagree
0:
N/A

Results
@DUX_YooYoung
Overall
Task
Performance
(Summary)
T1:
On
the
Library
website,
find
information
on

how
many
books
you
can
borrow.
Data
sets:
All
64
20
53
23
0
10
20
30
40
50
60
70
Y N NC SK
#
of
students
Y:
Right
answer
N:
Wrong
answer
NC:
Non
complete
SK:
Skip

Results
@DUX_YooYoung
Overall
Task
Performance
(Behavior)
T1:
On
the
Library
website,
find
information
on
how
many
books
you
can
borrow.
Data
sets:
All
15%
85%
Search
vs.
Browse
S
B
0
5
10
15
20
25
30
35
40
45
SI SA SS SD
Search
0
50
100
150
200
250
300
BM BH BP BPI
Browse
#
of
times #
of
times
SI:
Search
IUCAT
SA:
Search
Articles
SS:
Search
Site
SD:
Search
Databases
BM:
Browse
Menus
BH:
Browse
Homepage
BP:
Browse
from
Previous
Tasks
BPI:
Browse
from
Previous
IUCAT

Results
@DUX_YooYoung
Overall
Task
Performance
(#
of
steps)
T1:
On
the
Library
website,
find
information
on

how
many
books
you
can
borrow.
Average:
3.4
steps
Data
sets:
students
who
got
right
or
wrong

answer.
Filter
by
Y
or
N
#
of
steps

Results
@DUX_YooYoung
Overall
Task
Performance
(seconds
spent)
T1:
On
the
Library
website,
find
information
on

how
many
books
you
can
borrow.
Average:
79.22
seconds
Data
sets:
students
who
got
right
or
wrong

answer.
Filter
by
Y
or
N
seconds

Results
@DUX_YooYoung
Overall
Task
Performance
T2:
On
the
Library
website,
find
your
subject

guide.
Data
sets:
All
91
8
57
4
0
10
20
30
40
50
60
70
80
90
100
Y N NC SK
#
of
students
Y:
Right
answer
N:
Wrong
answer
NC:
Non
complete
SK:
Skip

Results
@DUX_YooYoung
Overall
Task
Performance
(Behavior)
T2:
On
the
Library
website,
find
your
subject
guide.
Data
sets:
All
13%
87%
Search
vs.
Browse
S
B
0
2
4
6
8
10
12
14
SI SA SS SD
Search
0
20
40
60
80
100
120
BM BH BP BPI
Browse
#
of
times #
of
times
SI:
Search
IUCAT
SA:
Search
Articles
SS:
Search
Site
SD:
Search
Databases
BM:
Browse
Menus
BH:
Browse
Homepage
BP:
Browse
from
Previous
Tasks
BPI:
Browse
from
Previous
IUCAT

Results
@DUX_YooYoung
Overall
Task
Performance
(#
of
steps)
T2:
On
the
Library
website,
find
your
subject

guide.
Average:
1.8
steps
Data
sets:
students
who
got
right
or
wrong

answer.
Filter
by
Y
or
N
#
of
steps

Results
@DUX_YooYoung
Overall
Task
Performance
(seconds
spent)
T2:
On
the
Library
website,
find
your
subject

guide.
Average:
43.09
seconds
Data
sets:
students
who
got
right
or
wrong

answer.
Filter
by
Y
or
N
seconds

Results
@DUX_YooYoung
Overall
Task
Performance
T3:
On
the
Library
website,
find
information

about
the
3D
Printing
Studio.
Data
sets:
All
84
11
60
5
0
10
20
30
40
50
60
70
80
90
Y N NC SK
#
of
students
Y:
Right
answer
N:
Wrong
answer
NC:
Non
complete
SK:
Skip

Results
@DUX_YooYoung
Overall
Task
Performance
T4:
On
the
Library
website,
where
would
you
go
when
you
don’t
know
where
to
start
on
your
research
project?
Categories #
of
students
NC 61
SK 15
Ask
Us 5
UL
Main
Web 2
A-‐Z
Databases 7
How
to
Guides 25
Subject
Guides 16
IUCAT 7
Reference
Collections 7
Find… 8
Library
tips 1
E-‐journal
homepage 1
Irrelevant 5

Results
@DUX_YooYoung
Overall
Task
Performance
T5:
Using
the
Library
website,
find
one
scholarly

article
in
your
major.
Data
sets:
All
39
43
62
16
0
10
20
30
40
50
60
70
Y N NC SK
#
of
students
Y:
Right
answer
N:
Wrong
answer
NC:
Non
complete
SK:
Skip

Results
@DUX_YooYoung
Overall
Task
Performance
(Behavior)
T5:
Using
the
Library
website,
find
one
scholarly
article
in
your
major.
Data
sets:
All
35%
65%
Search
vs.
Browse
S
B
0
5
10
15
20
25
30
35
40
45
50
SI SA SS SD
Search
0
10
20
30
40
50
60
70
80
90
BM BH BP BPI
Browse
#
of
times #
of
times
SI:
Search
IUCAT
SA:
Search
Articles
SS:
Search
Site
SD:
Search
Databases
BM:
Browse
Menus
BH:
Browse
Homepage
BP:
Browse
from
Previous
Tasks
BPI:
Browse
from
Previous
IUCAT

Results
@DUX_YooYoung
Overall
Task
Performance
(Keywords)
T5:
Using
the
Library
website,
find
one
scholarly

article
in
your
major.

Results
@DUX_YooYoung
Overall
Task
Performance
T6:
Using
the
Library
website,
find
one
database

in
your
major.
Data
sets:
All
68
26
61
5
0
10
20
30
40
50
60
70
80
Y N NC SK
#
of
students
Y:
Right
answer
N:
Wrong
answer
NC:
Non
complete
SK:
Skip

Results
@DUX_YooYoung
Overall
Task
Performance
(#
of
steps)
T6:
Using
the
Library
website,
find
one
database

in
your
major.

Results
@DUX_YooYoung
Difference
between
what
they
said
and
what
they
did?
0
1
2
3
4
5
0 1 2 3 4 5
v-‐Y
p11
Confident
&
Right
answer
v-‐Y Predicted
v-‐Y Linear

(Predicted
v-‐Y)
0
1
2
3
4
5
0 1 2 3 4 5
v-‐Y
p12
Frustrated
&
Right
answer
v-‐Y Predicted
v-‐Y Linear

(Predicted
v-‐Y)
Scores Scores
5:
Strongly
agree
4:
Agree
3:
Neither
agree
nor
disagree
2:
Disagree
1:
Strongly
disagree
0:
N/A

Results
@DUX_YooYoung
The
Role
of
Library
Instruction
in
DUX
0
1
2
3
4
5
Easy
to
use
Fun
to
use
Helpful
User-‐friendly
Confident
Satisfied
Positive
Aspects
Before After
0
1
2
3
4
5
Difficult
to
use
Frustrating
to
use
Useless
Confusing
Frustrated
Uncertain
Negative
Aspects
Before After
α =
0.05
5:
Strongly
agree
4:
Agree
3:
Neither
agree
nor
disagree
2:
Disagree
1:
Strongly
disagree
0:
N/A

Discussion
&
Next
Steps
@DUX_YooYoung
1. One
search
box
• Although
browse
is
still
dominant,
three
search
boxes
are

not
ideal…
If
students
decide
to
search,
students
attempt

to
use
IUCAT
first
without
notice.
• Solution
=>
QuickSearch from
North
Carolina
State

University
Library

Discussion
&
Next
Steps
@DUX_YooYoung
2. How
to
make
the
UL
library
website
fun
for
learning?
• Low
point
still
after
library
instruction
3. How
to
bring
library
instruction
to
web
design?
• Solution:
Working
with
Educational
Services
Charter
Group

(ESCG)
• ESCG
develops
more
fine-‐tuned
and
specialized
tutorial

guides

Discussion
&
Next
Steps
@DUX_YooYoung
4. How
to
maintain
students’
information
literacy
skills
after
library

instruction?
How
to
design
UL
website
to
make
students
better

information
literate?
• Solution
-‐ Start
with
LibGuides:
Pathfinder
vs.
Pedagogical
with

Educational
Development
Librarian

Challenges
@DUX_YooYoung
1. Library
Instruction
• Time
limit:
some
of
students
were
late,
so
they
missed
the
tasks

and
pre-‐survey
=>
non-‐valid
data
• Professor?
Instructor?
2. Data
coding
• 160
pre
&
post
survey
data
• 160
video
to
be
analyzed
3. We
knew
what
they
did,
but
didn’t
know
why
they
did
it

Journey from subjective to objective: Capturing user experience

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Journey from subjective to objective: Capturing user experience (20)

Recently uploaded (20)

Journey from subjective to objective: Capturing user experience