Pulse UX Testing: Bring the user into the user story. A.Witteman&R.vandenOever

Copyright © A. Witteman & R. van den Oever, All Rights Reserved.
Bringing the user into the
user story
Pulse Testing by Anna Witteman & Robin van den Oever
Mobile App Europe - Sep 2014

Today
What we will talk about

Bringing the user into the user story
helped us deliver better products
faster.

2002 2007 2014

Requirements
• Requirement 1
• Requirement 2
• Requirement 3
• Requirement N

‘As a user I want to
..................................
so that I can
................................’

• deliver a better product, faster
• strengthen your team
• with hardly extra costs
Giving the user a place in the process, will

Introduction
Who we are and where we
are from?

Hello :)
Robin van den Oever
Project manager/Agile coach
robin@icemobile.com
Anna Witteman
Head of UXLab
anna@icemobile.com

• Amsterdam based company
• 100 employees
• over 25 nationalities
• Either a foodie, code king, design nerd, beer brewer or unicorn
IceMobile Amsterdam

the full mobile agency

App icons
the mobile agency for food retail

Products
Specialized in Food Retail
Focus on (end) Users
One-offs
All round
Focus on clients

Problem
Why don’t we test frequently
with users?

“It doesn’t
fit in our
sprint.”
“The results
are hard to
measure.”
“It feels
like extra
work.”
“We were
just up
to speed.”
“It is too
soon to test
with users.”
“We don’t have
time to make
a prototype.”
“Knowing the
user is not
part of my job.”
“How do I know
what I will get
out of it?”
“Where do
we get
users from?”
“How do we know
which test results
we should use?”
“We do not have
resources to do
user tests.”

Interest:
Plan: How and when are we doing user testing?
Is it really a plan or more an improvisation?
Means:
Interest: What is the motivation for user testing?
Improve the product or prove you are right?
Plan:
Vision: Does decision-makers really have a good UX as a priority?
Or is more a buzz word?
Interest:
Vision
Interest
Plan
Means
Skills
What do we need to be able to start user testing?
Users? Prototype? Software? etc.
Skills:Can we conduct a user test?
Do we know how?

Transfer information effective
The team should learn most of the user test, not the UX researcher
Fit the user into the process
Do testing so that it is not ‘extra work’
Make results visible
Value should be clear and measurable
We wanted to find a way to:

Our solution
How to create a place where
great UX will happen.

Which practice proved its value?
Individuals and interactions
Working software
Customer collaboration
Responding to change
over
over
over
over
processes and tools
comprehensive documentation
contract negotiation
following a plan
Agile Manifesto

Setup
Process
Proof

Applying the agile principles to user testing
• Give team members responsibility
• Reduce paperwork to the minimum
• Giving the team little overhead
• Reuse documentation written from an user perspective

• Write a scenario
• Create a prototype
• Recruit users
• Pilot test
• Conduct Tests
• Edit videos
• Analyze video
• Write report
• Present results

Product = Prototype
• Write a scenario
• Create a prototype
• Recruit users
• Pilot test
• Conduct Tests
• Edit videos
• Analyze video
• Write report
• Present results
User test = Report

Our setup

More evaluators = more issues identified
Source: Herttzum, M., Jacobsen, N. (2001). The evaluator’s effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, vol. 13, pp.421-443.

Ratio benefits to costs of # evaluators
4 = ideal
#
evaluators
Source: Herttzum, M., Jacobsen, N. (2001). The evaluator’s effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, vol. 13, pp.421-443.

Conclusion
• Optimized the user test:
• Give the team responsibility
• No paperwork and no overhead
• Maintain a mutual goal
• Make the team smarter over time
• Team gets motivated

Setup
Process
Proof
Setup
Process
Proof

• Divided total work in small iterations
• Recurring rituals
• Easy to change course if required
• Give room to improve the process

Our agile way of working
Product
Backlog
Sprint
Backlog
Potentially
Shippable
Product
2-3
weeks
24
hours
Daily stand up
Design
Development
Testing

Source: Nielsen, Jakob, and Landauer, Thomas K.: "A mathematical model of the finding of usability problems," Proceedings of ACM INTERCHI'93 Conference (1993), pp. 206-213.
How many users should do a usability test?

Iterative Testing
•Prioritize: test important part(s) most extensive
•Frequent: reliable results, also test improvements
•Empathize: learn what matters to users
•Optimize: learn to test
Allow responding to change
'I have not failed.
I've just found 10000
ways that won't work.’
Sir Thomas Edison

iteration is the heartbeat
of agile & usertesting
use it!

Our renewed agile way of working
Product
Backlog
Sprint
Backlog
Potentially
Shippable
Product
2-3
weeks
24
hours
Daily Scrum
Meeting
Design
Development
Testing
User Testing

• User test in every sprint
• User test with 4 users
• Scenario about sprint (& previous sprints)
• No need to react immediately to findings
Pulse Testing: bring the user into the Sprint

• Always show progress visually
• Describe requirements from a user perspective (as a user... so that..)
• Discuss all work created with each other being responsible as a group

• Issues
• Insights
• Wish list
• Fixed issues (adjusted issues)
Make all results tangible

How to keep track of the findings?
# user that
did this task
# user that had
this problem
time stamps
of this issue
on video
description
of issue

How to keep track of the findings?

Follow up and integrate in existing process:
1.The most important issues will become user stories itself
2.These user stories are put in the product backlog
3.Most important issues will surface over time

Start tracking your Sprint contents
• Because you want to understand the impact of user testing
• Identify types of user stories in sprint:
• New Features
• Bugs
• Rework resulting from user testing

Example sprint contents
0
15
30
45
60
1 2 3 4 C 5 6 7 8 C
New Features Bugs Rework resulting from UT

Back to you.
What it comes down to.

• Look at common practices within your company
• Eliminate waste
• Merge iterations
• Visualize progress
Learnings

Every project has its own
challenges...
the most important
thing is to start,
you will make it work
for you!

How are we and where are we from?Thank you!
Questions? Feedback? Share experience with user testing?
Please contact us: anna@icemobile.com & robin@icemobile.com
We love to stay in contact!

http://instagram.com/icemobile
https://www.facebook.com/IceMobileAmsterdam
https://twitter.com/IceMobile
http://icemobile.com/blog
https://linkedin.com/company/icemobile
Follow us and like us on:

The mobile agency for food retail

Pulse UX Testing: Bring the user into the user story. A.Witteman&R.vandenOever

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to Pulse UX Testing: Bring the user into the user story. A.Witteman&R.vandenOever (20)

More from Anna Witteman (8)

Recently uploaded (20)

Pulse UX Testing: Bring the user into the user story. A.Witteman&R.vandenOever