UX Sofia 2012 - DIY Mobile Usability Testing Workshop

Bernard, packet core
engineer at NSN

Belén, interaction
designer at Intel’s OTC

how to plan and run usability
tests with mobile devices in a
laboratory setting

assuming ...
1. you have experience planning and running
usability tests with desktop computers

2. you are testing software or an interactive
prototype running on a mobile device

3. you are testing in a usability laboratory or
similar environment

we will not focus on
1. the fundamentals of the usability testing
technique

2. how to prototype for mobile devices

3. how to test in the ﬁeld

these are
the golden
rules you
need to
remember

usability testing
a process that employs people as testing
participants who are representative of the
target audience to evaluate the degree to
which a product meets speciﬁc usability
criteria.

Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell

usability testing
watching people try to use what you’re
creating/designing/building (...) with the
intention of (a)making it easier for people to
use or (b)proving that it is easy to use.

Rocket Surgery Made Easy, S. Krug

Desktop Mobile
Product
software software

Mobile
Device Computer
phone

Context Predictable Unpredictable

Mobile phone
Connection DSL / Cable
network

which phone?
which context?
which connection?

web task success rates
feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009
http://www.useit.com/alertbox/mobile-usability-study-1.html

handset usability
affects test results

test with
participants’
own phones

when not possible,
include training and
warm-up tasks

handset becomes a
screening criteria when
recruiting test subjects

http://goo.gl/hmp8t

mobile phone subscriptions per 100 people

http://data.worldbank.org

http://gs.statcounter.com/

WP iOS Android Blackberry

http://developer.android.com/resources/dashboard/platform-versions.html

http://us.blackberry.com/developers/choosingtargetos.jsp

do not
confuse
target
platform
with target
user

which phone would you use for testing?
who are the target users?
handset screening criteria
will you need warm-up tasks? (yes / no)
if yes, which ones?

BNT are considering a
mobile version of their
website.

Before investing in the
project, they would like to
know how their existing
desktop site performs on
mobile devices.

You are working on an
Android application for
Soﬁa Airport.

Together with your
developers, you have
created a ﬁrst version of
the app. This ‘alpha’
version is not available in
Google Play, it is not fully
tested and might be
buggy.

St. Ekaterina University
Hospital has developed a
Blackberry application
that helps nurses and
doctors access and
update patients records
during their daily shift.

They have asked you to
help them test the
application.

Evernote wants to
uncover the main usability
issues in their existing
mobile applications for
iOS, Android, Windows
Phone and Blackberry.

Help them organise the
necessary testing.

what about emulators?

mas_fsx_ by Massimiliano Silipigni at http://www.ﬂickr.com/photos/silipigni/6055803785/

good for ...
using an emulator enables testers to
thoroughly capture user behaviour (...) The
captured data are generally informative and
useful for analysing user performance

Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications,
D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)

good for ...
emulators are more suitable to be used for
improving the interface design of applications
such as the layout of menu structures during
the development process.


but bad for ...
using emulators alleviates the problems of
latency, inefﬁcient input mechanisms, and the
changing wireless environment, potentially
leading to untruthful user satisfaction


ﬁeld vs. lab

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field
C.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

ﬁeld vs. lab

0 0

The results show that the added value of
conducting usability evaluations in the ﬁeld is
very little and that recreating central aspects of
the use context in a laboratory setting enables
the identiﬁcation of the same usability problem
list.
Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware
Mobile Systems in the Field
J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

0 1
The results show that the added value of
conducting usability evaluations in the ﬁeld is
very little and that recreating central aspects of
the use context in a laboratory setting enables
the identiﬁcation of the same usability problem
list.

Field Lab
Mobile Systems in the Field
J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

according to our study there was no difference
in the number of problems that occurred in the
two test settings. Our hypothesis that more
problems would be found in the ﬁeld was not
supported
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing
A. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

0 2
according to our study there was no difference
in the number of problems that occurred in the
two test settings. Our hypothesis that more
problems would be found in the ﬁeld was not
supported

Field Lab

evaluations conducted in ﬁeld settings can reveal
problems not otherwise identiﬁed in laboratory
evaluations

1 2
evaluations conducted in ﬁeld settings can reveal
problems not otherwise identiﬁed in laboratory
evaluations

Field Lab

The analyses of the comparison between
usability testing done in two different settings
revealed that there were many more types and
occurrences of usability problems found in the
ﬁeld than in the laboratory. Those problems
discovered tend to be critical issues.
Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field Tests
H.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006

2 2
The analyses of the comparison between
usability testing done in two different settings
revealed that there were many more types and
occurrences of usability problems found in the
ﬁeld than in the laboratory. Those problems
discovered tend to be critical issues.

Field Lab
Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field Tests
H.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006

ﬁeld vs. lab

D ISA GR EE
EXPE RTS

... but they all agree

testing in the ﬁeld requires double the time
in comparison to the laboratory


evaluations in the ﬁeld (are) more complex
and time-consuming


ﬁeld-based usability studies are not easy to
conduct. They are time consuming and the
added value is questionable.
Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

testing in
the lab is
better than
no testing

can you think of an
example of mobile
software that you
must test in the ﬁeld?

if doing ﬁeld testing ...
do it late
plan an run pilot tests
be prepared (like the Scouts)

if doing ﬁeld testing ...
have I chosen an appropriate location?
which external factors could impact results?
is the equipment working as planned?
do I have enough time to run the tests?
does the protocol work? Is it well
understood?

If we cannot go to the ﬁeld,
the ﬁeld must come to us

Bringing the ﬁeld into the lab

Simulations
Scenarios
Take the lab out

simulations
Use of techniques to “reﬂect or recreate a
mobile use situation” (...) their level of
sophistication is measured by the extent to
which the environmental and ergonomic
factors of the actual use context are
recreated.
Emerging Research Methods for Understanding Mobile Technology Use,
P. Hagen, T. Robertson, M. Kan and K. Sadler, Proceedings of OZCHI 2005, Canberra (Australia)

Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile
Systems in the Field, J. Kjeldskov, M.B. Skov, B.S. Als and R.T.Høegh, MobileHCI 2004

Experimental Evaluation of Techniques for Usability Testing of Mobile Systems in a Laboratory Setting, E.T.
Beck, M.K. Christiansen, J. Kjedskov, N. Kolbe and J. Stage, OZCHI 2003, Brisbane (Australia)

scenarios

First you choose the tasks to test (...) Then
you expand these tasks into scenarios (...) A
scenario provides some context and
supplies information the user needs to know


a task:
Find a hotel

a desktop scenario:
You are planning a surprise weekend trip to
Paris. You’ll be leaving on Friday and coming
back on Sunday evening. Your partner’s
favourite Paris area is Le Marais. Find a place
to stay he/she will like.

a task:
Find a hotel

a mobile scenario:
You were late to Orly airport and have
missed your ﬂight home. There are no more
ﬂights from Paris today, so you’ll have to
spend the night. Since it’s already 10 pm, use
your phone to look for a place to stay close
to the airport.

portable labs can be
deployed anywhere
document
camera

webcam

laptop

Jyske Bank in Roskilde by king_david_uk at http://www.ﬂickr.com/photos/kingdavid/2926673466/

Vodafone, Croydon, London CR0 by Kate Pugh at http://www.ﬂickr.com/photos/kake_pugh/6545344321/

write a mobile
scenario for your
group’s task

You are testing:

http://www.tﬂ.gov.uk

Task:

Find a public transport route

A bit of context:

From Heathrow Terminal 5 to the Olympic Stadium.
User must arrive on time for the Olympics opening
ceremony (27th July at 7:30pm)

You are testing:

http://www.toptable.co.uk

Task:

Find a restaurant

A bit of context:

Lebanese cuisine
Around Shoreditch (an East London area)
For tomorrow at 8:30pm
For 5 people

You are testing:

An Android app called ‘CycleStreets’, which helps you
ﬁnd cycling routes across London

Task:

Calculate a route distance

A bit of context:

From King’s Cross station to Tate Modern
Route with least trafﬁc

You are testing:

A task management app called Remember the Milk

Task:

Create a task

A bit of context:

The task is to buy ﬂowers for a friend (make up a name)
Next Tuesday morning
In her favourite ﬂower shop, called Blooming

how often are mobile
users connecting their
devices to WiFi
networks instead of
mobile networks?

Towering by Rachel Strohm at http://www.ﬂickr.com/photos/rachelstrohm/4693016049/

25% of households worldwide have WiFi networks
set up
Globally, smartphones account for 36% of all public
WiFi hotspot connections, compared to 48% for
laptops and 10% for tablets
Out of 4 billion unique mobile phone users, 1
billion uses WiFi to access the Internet (there are
only about 900 million smartphones)

http://www.lukew.com/ff/entry.asp?1537

http://communities-dominate.blogs.com/brands/2012/05/mobile-services-beyond-
messaging-excellent-tns-global-survey-reveals-tons.html

http://databank.worldbank.org

cover your
participants’
data costs

remote?

Remote Mania by Dramagirl at http://www.ﬂickr.com/photos/dramagirl/2268235917/

http://www.usertesting.com/mobile

but ...
I’d say remote testing gives you about 80% of
the beneﬁts of a live test with about 70% of
the effort. You do lose 20% of something not
being in the same room as the participant.
The in-person experience is just richer,
somehow.

to record or
not to record ...

To be or not to be, that is the question by Mister_Jack at http://www.ﬂickr.com/photos/machineman22/5130901135/

it is difﬁcult ...
We have used Morae in the past also for
website testing, but could not easily use the
system for mobile (...) Obviously it’s difﬁcult
to video something like that so we just sit
next to them and observe.

http://belenpena.posterous.com/user-research-for-mobile-software-a-short-sur

... but video is both a
memory aid
powerful communication tool

which phone?
which context?
which connection?
which recording setup?

Mobiola Screen Capture for Blackberry 4.2+ and Symbian S60 v3
http://www.shapeservices.com/en/products/details.php?product=capture&platform=none#

USB webcam records document camera
participant’s face records mobile screen

both cameras connect to a document camera range
laptop running screen is marked with tape
recording software

Mobile Device Camera by Noldus Ovo Studios device camera
http://www.noldus.com/human-behavior-research/ http://www.ovostudios.com/devicecamera.asp
accessories/mobile-device-camera-mdc#

Little Springs Design
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/
Google
Towards the Perfect Infrastructure
for Usability Testing on Mobile
Devices, R. Schusteritsch, C.Y. Wei,
M. LaRosa - Google (CHI 2007)
Usability Sciences
http://www.usabilitysciences.com/services/lab-
based-usability-testing/mobile-usability-testing

Nick Bowmast http://www.bowmast.com/mob-device-cam/

are blooming!!

Little Springs Design Nick Bowmast http://www.bowmast.com/mob-device-cam/
http://www.littlespringsdesign.com/blog/2008/Jun/usability-
testing-for-mobile-devices-2/

Google
Towards the Perfect Infrastructure
for Usability Testing on Mobile Usability Sciences
Devices, R. Schusteritsch, C.Y. Wei, http://www.usabilitysciences.com/services/lab-
M. LaRosa - Google (CHI 2007) based-usability-testing/mobile-usability-testing

the observation room
screen audio
Skype Skype

Screen sharing
Telephone conferencing
system
Mirroring

Mirroring

Screen sharing

Telephone conferencing system

screen capture applications
pros cons
participants won’t appreciate
you installing stuff on their
phones
provides high quality
screen recording no application will
support all platforms
in touchscreen phones, ﬁnger
interactions are not captured

Ovo Studios screen capture application for iOS
http://www.ovostudios.com/

Obviously,
think aloud is critical because I cannot see how the participant is interacting
with his fingers on the touch screen. ... and that
is a big
I can see using the mirroring in a lab setting to get the signal from the iPad
to the observation room though you still won’t see the physical interaction
deal
with the device, like you would with a device mounted camera (e.g. Noldus).

Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011 - http://www.remoteresear.ch/airplay/

Recently, we’ve questioned the value of capturing the device screen. It does
after all end up being a video of the screen changing but with no sense
of the participant interacting with it.

... and that
is a big
deal

iPad usability testing - our equipment - http://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/

document cameras
pros cons
document cameras
good recording quality
bring up the price
participants must keep within
easy to set up
the camera range
phone must lay on a desk or
be hold at a ﬂat angle

mounted devices
pros cons
ready-made devices will bring
up the price
allow natural interaction with if bulky, they can prevent
the phone single-hand use
if heavy, they can become
tiring during long tests

the ideal recording setup
easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & ﬁngers
gives enough video quality

screen mounted
the ideal recording setup capture
applications
document
cameras
devices
ready-made
mounted
devices DIY

easy to put together
cheap
repeatable
allows holding the device
allows one-handed use
supports all form factors
runs tests with participants’ phones
captures screen, face & ﬁngers
gives enough video quality

no recording
setup is
perfect: just
use the best
tool for the
job

test with participants’ own phones
do not confuse target platform with target user
testing in the lab is better than no testing
do not test over WiFi
cover your participant’s data costs
no recording setup is perfect: just use the best
tool for the job

thanks!!!
www.DIYMobileUsabilityTesting.net

UX Sofia 2012 - DIY Mobile Usability Testing Workshop

More Related Content

Similar to UX Sofia 2012 - DIY Mobile Usability Testing Workshop (20)

More from Belen Barros Pena (7)

Recently uploaded (20)

UX Sofia 2012 - DIY Mobile Usability Testing Workshop