What Does it Mean for a System to be Useful? An Exploratory Study of Usefulness

What Does it Mean for a
System to be Useful?
An Exploratory Study of Usefulness
Craig M. MacDonald, Ph.D.
Pratt Institute, School of Information & Library Science
Michael E. Atwood, Ph.D.
Drexel University, College of Computing & Informatics
25 June 2014
Designing Interactive Systems | Vancouver, BC2

HCI has evolved
2
Web, mobile, and personal technologies
have drastically altered how, where, and
why people use computers.
(Bødker, 2006)
User Experience (UX), which treats human-
computer interaction as a cognitive-
emotional process, is emerging as a new
paradigm for the field.
(Bargas-Avila & Hornbæk, 2011)
"The 21st Century Concert Experience" by Al Case is licensed under CC BY-
NC-ND 2.0

Usability – is it the only factor?
3
"Tricycle" by Aslak Raanes is licensed under CC BY 2.0
“If ease of use was
the only valid
criterion, people
would stick to
tricycles and never
try bicycles.”
- Douglas Engelbart

Usability = the gold standard
Usability research has dominated HCI over the
past 40 years.
It has yielded tremendous insights into how to
design systems that are easy to use and easy to
learn.
But, little time has been spent on determining
whether systems are actually useful.
4
Source(s): Hornbæk (2010); MacDonald & Atwood (2013)

Usefulness is not new…
Designing useful systems has long been cited as one of
the primary goals of user-centered design.
5

…but it’s mostly a mystery
Studies of usefulness are largely absent from
HCI research.
There is no empirical evidence about the relationships
between usefulness and other factors.
There is no widely accepted deﬁnition of
usefulness in HCI.
When people talk about usefulness, we can’t be certain
they’re talking about the same thing.
This research aimed to address these two areas by
(1) defining usefulness and (2) studying
usefulness in a controlled experiment.
6

So, what is usefulness?
Many different definitions or uses of the term
usefulness, each of which covered one or more
of the following:
1.  The functions provided by the system.
e.g., “the functions people really need in their
work” (Gould & Lewis, 1985)
2.  The tasks users are trying to complete.
e.g., “the role of the technology in accomplishing the
user’s relevant tasks” (Maryniak-Nelson & Caldwell, 1992)
3.  The goals users are trying to achieve.
e.g., “whether system can be used to achieve some
goal” (Nielsen, 1993)
4.  The context in which the system is being used.
e.g., “how technology can fit into users' actual social and/
material environments” (Nardi, 1996)
7

A working deﬁnition:
Usefulness is “the extent to which a system’s
functions allow users to complete a set of
tasks and achieve specific goals in a
particular context of use.”
This may sound similar; according to the ISO:
Usability is “the extent to which a product can be
used by specified users to achieve specified
goals with effectiveness, efficiency, and
satisfaction in a specific context of use.”
8

What is usability?
There is surprisingly little consensus about a
precise definition of usability. (van Welie, van der Veer,
& Eliëns, 1999)
But there is broad agreement that usability
refers to effectiveness, efﬁciency, and
satisfaction. (Chen, Germain, & Rorissa, 2009; Hornbæk
2006; Hornbæk & Law, 2007; ISO 9241-11; Nielsen, 1993)
9

Our (imperfect) distinction
Usefulness
a system’s appropriateness for a specific context
Usability
its effectiveness, efficiency, and satisfaction within
that context
10

Method
We designed and conducted an exploratory
experiment* to:
1.  examine the effects of context on usefulness;
and
2.  explore the relationships between usefulness
and three other UX attributes: usability,
aesthetics, and enjoyment.
*Yes, a laboratory study of context seems like an oxymoron but we believe
it is a good starting point for studying usefulness.
11

What is context?
Many ways of defining context; for this study,
context is defined simply as:
12
USER
TASK
TOOL
ENVIRONMENT
Source(s): Suchman (1987); Bannon (1990); Schilit & Theimer (1994); Hutchins (1995); Kuutti (1995); Nardi (1996);
ISO (1997); Dey, Abowd, & Salber (2001); Dourish (2004); Räsänen & Nyce (2006); Connolly, Chamberlain, & Phillips
(2008)
(diagram adapted from Shackel, 1991)

Controlled: User
The user dimension can depend on many
possible sub-dimensions.
So, we tried to control for this factor by
limiting the study population to a group likely
to share common interests, knowledge, and
interaction styles/preferences.
13

Independent Variable: Task-Env
Three written scenarios: one “no scenario” as a
control and two scenarios that differed along
the task and environment dimensions:
14
Task-Environment 1 Task-Environment 2
Task:
EXPLORATION (action mode): find
reputable resources with facts, pictures,
charts/graphs, etc., to include in
presentation about Climate Change.
Task:
RETRIEVAL (goal mode): find whether
the name of the first computer was
“IPECAC” and whether it was invented in
1928 at Drexel University.
Environment:
Early on a weekday morning
In the library computer lab
Class assignment
Environment:
Late on a weekday evening
In a student dormitory room
A wager with a friend

Independent Variable: Tool
15
Educational information portals
were chosen because they:
1.  offer a range of potential use
cases;
2.  are relatively easy to learn;
and
3.  are familiar to the study
population.

Dependent Variables
Variable Scale Items Source
USEFULNESS 1.  The website provides the right functions.
2.  I am able to use the website to complete my task(s).
3.  I am able to use the website to fulfill my goal(s).
4.  The website fits my current situation.
5.  The website is useful.
Developed for this
research
USABILITY 6.  The website is easy to use.
7.  I feel in control when I am using this website.
8.  The website requires little effort to use.
9.  Using the website is effective.
De Angeli,
Hartmann, &
Sutcliffe (2009)
AESTHETICS 10.  Everything goes together on this website.
11.  The color composition is attractive.
12.  The layout appears professionally designed.
13.  The layout is pleasantly varied.
Moshagen &
Thielsch (2010)
ENJOYMENT 14.  I find using the website to be enjoyable.
15.  The actual process of using the website is pleasant.
16.  I have fun using the website.
van Shaik & Ling
(2011)
GOODNESS 17.  I judge the website to be: bad—good Hassenzahl (2004)
16

Study Design
We used an incomplete repeated measures
design in which participants were exposed to
all three levels of each variable exactly once.
17

Post-Study Questionnaire
First, participants responded verbally to the
following question:
What makes a website useful to you? What criteria
do you look for?
Next, participants answered questions
regarding:
Experience with the websites used in this study
Knowledge of the Internet (iKnow) (Potosky, 2007)
Basic demographic information (age, gender)
Background (major, class level, number of HCI
courses)
18

Participants
36 undergraduate students enrolled at Drexel
University in the Information Technology,
Information Systems, or Software Engineering
programs.
All sophomores or above.
Nearly all (35; 97%) had taken at least one HCI course.
Other characteristics:
Median age range: 21 years old
Gender: 86% male, 14% female
iKnow (0-70): ranged from 52 to 70 (average = 62.81)
19

Data Analysis: Marginal Models
A special case of Linear Mixed Models that
include:
Fixed effects: can be interpreted similarly to
ANOVA
Estimates of regression coefficients: can be
interpreted similarly to traditional regression
analysis
20

Parameter Estimate t Sig.
USABILITY 1.386 16.834 < 0.001**
AESTHETICS -0.633 -5.613 < 0.001**
AESTHETICS * Climate Ch. 0.472 3.600 0.001**
AESTHETICS * First Comp. 0.100 0.836 0.405
AESTHETICS * Control 0.000 . .
iKNOW * Climate Ch. -0.025 -2.986 0.004**
iKNOW * First Comp. 0.002 0.370 0.712
iKNOW * Control 0.015 2.342 0.021*
Marginal Model: USEFULNESS
21
Source F Sig.
USABILITY 283.391 < 0.001
AESTHETICS 29.856 < 0.001
TASK-ENV * AESTHETICS 6.961 0.001
TASK-ENV * iKNOW 5.513 0.002
Final Marginal Model
Coefficient Estimates
Higher ratings of USABILITY
associated with higher ratings of
USEFULNESS.
Higher ratings of AESTHETICS
associated with lower ratings of
USEFULNESS (but not as much
under Climate Change scenario).
Higher iKNOW scores associated
with lower ratings of USEFULNESS
under Climate Change scenario.
Higher iKNOW scores associated
with higher ratings of USEFULNESS
under Control scenario.

Marginal Model: GOODNESS
22
Parameter t Sig.
TOOL: ipl2 -0.557 0.580
TOOL: RefSeek 2.559 0.013*
TOOL: Awesome Lib. -2.544 0.013*
USEFULNESS 5.464 0.000**
USABILITY -0.241 0.810
USEFULNESS * Climate Ch. 1.036 0.303
USEFULNESS * First Comp. -3.667 0.001**
USEFULNESS * Control . .
AESTHETICS * Climate Ch. 4.600 < 0.001**
AESTHETICS * First Comp. 5.699 < 0.001**
AESTHETICS * Control 0.684 0.497
ENJOYMENT * Climate Ch. -3.071 0.003**
ENJOYMENT * First Comp. 0.898 0.372
ENJOYMENT * Control 2.850 0.006**
iKNOW * ipl2 0.015 0.988
iKNOW * RefSeek -2.266 0.027*
iKNOW * Awesome Lib. 2.589 0.012*
USABILITY * ipl2 4.783 < 0.001**
USABILITY * RefSeek 2.804 0.007**
USABILITY * Awesome Lib. . .
AESTHETICS * ipl2 -4.085 < 0.001**
AESTHETICS * RefSeek -3.309 0.001**
AESTHETICS * Awesome Lib. . .
Source F Sig.
TOOL 4.986 0.004
USEFULNESS 38.452 < 0.001
USABILITY 10.059 0.002
TASK-ENV * USEFULNESS 11.397 < 0.001
TASK-ENV * AESTHETICS 12.169 < 0.001
TASK-ENV * ENJOYMENT 6.350 0.001
TOOL * iKNOW 4.463 0.007
TOOL * USABILITY 12.015 < 0.001
TOOL * AESTHETICS 10.339 < 0.001
Final Marginal Model Coefficient Estimates
All variables had some effect on ratings of
GOODNESS, but USEFULNESS was the only
variable to have a significant relationship
across all systems and contexts.
Higher ratings of USEFULNESS were
associated with higher ratings of
GOODNESS under all three scenarios (but
the effect was weaker in the First Computer
scenario).

Qualitative Analysis
Four themes:
23
Appropriateness
to Context
Simplicity and
Ease of Use
Pleasurable
Interaction
Visual
Attractiveness
CODES:
- Suitable for Purpose/Goal
- Right Functionality
- Appropriate Content
CODES:
- Easy to Use/Navigate
- Speed/Efficiency in Use
- Organized/Uncluttered
- Streamlined/Simple Design
CODES:
- Pleasing to the Eye
- Craftsmanship
- General Attractiveness
CODES:
- Familiarity
- “It” Factor
- Irritation-Free
- Customizability
Usefulness
n=28
77.8%
n=17
47.2%
n=27
75.0%
n=17
47.2%

Usefulness is shaped by context
The quantitative analysis showed a significant
effect of contextual factors (the iKnow x Task-
Environment interaction) on perceptions of
usefulness.
The qualitative analysis revealed that many
participants defined a useful website in terms of
contextual factors (whether it provides the “right”
functions and access to the “right” information).
Conclusion: it is highly likely that the usefulness of
a system depends on the context in which it is
used.
24

Usability is linked to usefulness
The quantitative analysis showed that higher
ratings of usability were associated with
higher ratings of usefulness.
The qualitative analysis revealed that major
aspects of usability (efficiency/speed,
effectiveness, irritation-free, etc.) were
considered aspects of a useful website.
Conclusion: usability seems to be an integral
component of usefulness, but more data is
needed (especially at the boundaries).
25

Usefulness may be hedonic?
The quantitative analysis showed that websites
seen as more attractive were seen as less useful
and that usefulness was not influenced by
perception of enjoyment.
The qualitative analysis showed that almost 50%
of participants cited some aspect of aesthetics
and/or experience in their definitions of a useful
website.
Conclusion: the beauty dilemma1 is powerful, and
the experiential aspects of usefulness may be
more nuanced than “enjoyment.”
26
Source: 1 Diefenbach & Hassenzahl (2009)

Usefulness matters – a lot
The quantitative analysis showed that
usefulness was the only variable with a
significant effect on ratings of goodness
(regardless of contextual factors).
However, other factors (usability, aesthetics,
enjoyment) were also important depending on
the tool and/or task-environment.
Conclusion: usefulness is a necessary (but
maybe not sufficient) dimension of a good
system.
27

Implications for Research/Practice
1) Evaluators should explicitly address
issues related to usefulness.
Probing for usefulness is not uncommon, but it
is infrequent, informal, and without a
coherent connection to evaluation goals.
(Nørgaard & Hornbæk, 2006)
We encourage evaluators to incorporate
questions of usefulness into evaluation plans
and purposefully address usefulness during
user testing.
28

Implications for Research/Practice
2) Evaluators may consider varying
evaluation contexts.
Potential variants: random sampling of tasks,
allowing users to modify test environments,
holding tests in various locations, etc.
Doing so would jeopardize the validity of any
experiment, but there is plenty of evidence
that usability lab experiments are not valid
anyway. (Gray & Salzman, 1998; Lindgaard &
Chattrichart, 2007; Nørgaard & Hornbæk, 2006; )
29

Limitations & Future Work
This study was a laboratory experiment with a
relatively small and tightly controlled group of
mostly male tech-savvy undergraduate
students and a highly specific type of
interface.
We plan to address these limitations in future
work, and encourage other researchers to do
so as well.
30

Thank You
Craig M. MacDonald
cmacdona@pratt.edu
http://www.craigmacdonald.com
@CraigMMacDonald
31
Acknowledgements:
We thank Susan Weidenbeck, Michelle Rogers, Denise Agosto, and
Kasper Hornbæk for their guidance in shaping and directing this research

What Does it Mean for a System to be Useful? An Exploratory Study of Usefulness

More Related Content

What's hot (20)

Similar to What Does it Mean for a System to be Useful? An Exploratory Study of Usefulness (20)

More from craigmmacdonald (14)

Recently uploaded (20)

What Does it Mean for a System to be Useful? An Exploratory Study of Usefulness