Measuring What Matters: A UX Approach to Metrics :: UX Days Tokyo [April 2015]

Letʼs get to know each other
At each table...
* name
* role
* your favorite superpower
Hello!
5 min

@katerutter
tweet-friendly

I am not an expert on your business.
You are.
My goal is to help you explore
how to measure how well your product
serves real humans...
so you can use metrics to design & improve
the products you build.
What to expect...

What to expect...
We are going to be working fast.

What to expect...
There is a purpose to this.
I will push you to get something
down rapidly. In 1 day, youʼll learn how to
create actionable, UX-centered metrics to
track progress of your product.
We are going to be working fast.

What to expect...
You will be working solo and in pairs.
Note that these techniques are
very powerful in teams.
Weʼll keep talking to a minimum.
(Happy to talk more after the workshop.)

What to expect...
You may not like the pace/process/
pressure/perspective, etc. etc.
That is okay.
All I ask is that you keep an open
mind and try it.

How do you
measure UX
now?
10 min

UX + metrics
=
<3

•Identify and design for key interactions that
deliver measurable value to your users.
•Have a clear set of numbers that indicate
progress.
•Know what interactions you need to track to
measure user behaviors.
Why?

Why this matters:
“Over 10,000 downloads!”
“Avg time on site is 22 minutes!”
“+450 new sign-ins this week!”

Why this matters:
“+450 new sign-ins this week!”
FAIL

Why this matters:
“+450 new sign-ins this week!”is this good?

Product
(black box)
Conversion
Funnel
Trafﬁc

~ Alistair Croll
Don’t just ask questions. Know how
the answers to the questions will
change your behavior.

A good metric is...
Clear & speciﬁc
Comparative
Rate/Ratio
Changes your behavior

What we will make
Snapshot
What people?
What problems?
What product?
How will we
track progress?
Metric plan
What can a
user do with
our product?
Key use
What metric
matters?
Key metric

Make a
product
snapshot

Letʼs get warmed up
A product you work on
An idea for a new product
Pick an idea to
use for the day
2 min
Task-a-
Doodle
write it on a
stickynoteor

Create a Product Snapshot
Snapshot
What people?
What problems?
What product?
What can a
user do with
our product?
Key use
How will we
track progress?
Metric plan
What metric
matters?
Key metric

a mobile app for
delegating tasks
Working parents
with with kids
need to know when
something’s done
Too much to do, not enough time
need to share
tasks on the go
Task-a-doodle 4 19 2015
10 min
Who are the people?
(the users)
What problems do
they have?
What is a proposed
solution?
Product Snapshot
BE SPECIFIC

Is it clear?
Is it speciﬁc?
What can be
improved?
Product Snapshot
Share as a pair
5 min
(each)

What you have now...
Snapshot
What people?
What problems?
What product?
What can a
user do with
our product?
Key use
How will we
track progress?
Metric plan
What metric
matters?
Key metric

the
UX STACK

How do I
KNOW IT’S
WORKING?
I have a
product.
so...

Who is it for?
Why does it matter
to them?
What can they do that
they couldnʼt do before?
What features do they
need to do that?
How do those features
ﬁt together?
Users
Needs &
goals
Uses
Features
Product,
brand & voice
via @luxrco
UX elements

Users
Needs &
goals
Uses
Features
Product,
brand & voice
Features
Users
Uses
Product
Needs
via @luxrco
UX stack
via @luxrco

Users
Needs &
goals
Uses
Features
Product,
brand & voice
Features
Users
Uses
Product
Needs
Company
Purpose
via @luxrco
sketches,
prototypes,
pixels
UI
UX stack
via @luxrco

Envision a
Key Use

Envision a Key Use
Snapshot
What people?
What problems?
What product?
What can a
user do with
our product?
Key use
How will we
track progress?
Metric plan
What metric
matters?
Key metric

via @luxrco
Features
Users
Uses
Product
Needs
Company
Purpose
via @luxrco
sketches,
prototypes,
pixels
UI
UX stack

An example from Foodspotting
6-up uses

Envision 6 uses
sketch!
Task-a-doodle, Mary
4 19 2015

30 sec

Rapid Sketching
Star people!
People, Context & Emotion

You can draw a person in 4 simple steps....
Star People

Star People in context
Use simple
shapes to put a
person in a place.

smile frown
up brows
neutral
down
brows
no brows
add a mouth...
add eyebrows...
Expressions Matrix

2 min
Speed-sketch
10 star people!

Key Uses 6-up
OK to be obvious
Task-a-doodle, Mary
4 19 2015
•1 idea per panel
•6 different ideas
(not a sequence)
•Sketch AND write
What can someone
DO with your
product?
- Solves the problem
- Addresses a need
10 min

Key Use
•This is info transfer; not
a new idea
•Make the sketch clean &
clear
Select one and redraw.
5 min
Task-a-doodle,
Mary
4 19 2015
Share tasks
with others.
What can someone
DO with your
product?
OK to be obvious

Lunch!
http://www.g4tv.com/attackoftheshow/blog/post/715478/cutest-bento-boxes-for-geeks
1 hr

Identify a Key
Metric

Identify a Key Metric
Snapshot
What people?
What problems?
What product?
What can a
user do with
our product?
Key use
How will we
track progress?
Metric plan
What metric
matters?
Key metric

a good metric... a great metric...
...makes you look at all the
other metrics and say “none of
those other numbers matter if
we donʼt get this right ﬁrst.”
...measures the usage of your
product by a person. The usage
should be speciﬁc to features that
deliver value to your user.
vanity
total number
of registered
users
unhelpful
sign-ups
% of users
who share a
task 3+
times a day,
per week
awesomebetter
% of users
who sign in
3+ times a
day,
per week
good
% of new
users per
week
Metrics checklist

Brainstorm 5 ideas for
metrics
5 min
What could you count to make sure
the Key Use is happening?

Pick one to focus on
1 min

3. Is there an object
basis?
• “per user” [ ___/user]
• “per user per week” (fancy!)
• Examples: per user, per session,
or by interaction or item
(messages per ﬁle, emails
per thread)
* not required, but super-helpful
• “per week” [ ___/wk ]
• “per month” [ ___/mo]
2. Is there a time basis?
• “number of...” [ # ]
• “average number of...” [avg ]
• “percent of...” [ % ]
1. Does the metric
begin with a number?
Make it awesome 5 min

Share your work
Review the snapshot &
use.
Take 3 minutes to
conﬁrm clarity
Peers: make
recommendations
Share as a pair
5 min
(each)
Key Metric

10 min
Questions &
Discussion

% of users who wear a tie
within one week of receiving
it, per month.
% users who wear a received
tie 3+ times/week, per week.
% users who purchase a tie 3
shipments in a row, per week.

Put it on your dashboard
1 min

Pick another one
1 min

Break!
15 min

One more time
1 min

1 min
Pick ONE to work on next

Make a Plan

Make a Metric Plan
Snapshot
What people?
What problems?
What product?
What can a
user do with
our product?
Key use
How will we
track progress?
Metric plan
What metric
matters?
Key metric

Metrics Plan
•Set a goal
•Set a time span
•Think about how to
collect the data
How will you track
progress?
% of users
who share
a task 3+
times a day,
per week
Capture clicks on “share now”
- capture # of shares
- timestamp each
- capture by user ID
(dump into raw data file)
- use total # of users as base
1. Calculate by day, # of users with
3+ shares from 12:00am - 11:59pm
2. Use threshold of 3+ to order list
3. # of users above threshold / Total #
of users x 100 = %
Week starts Monday @ 12:00am,
ends Sunday @ 11:59pm
30%
8 1 2015
5%
4 19 2015

Metrics Plan
•Set a goal
•Set a time span
collect the data
How will you track
progress?
% of users
who share
a task 3+
times a day,
per week
- timestamp each
of users x 100 = %
30%
8 1 2015
5%
4 19 2015
- timestamp each
of users x 100 = %

Three ways to gather data:
• Non-technical
• Technical “by hand”
• Instrumented with software
Instrumentation
process of setting up the technology to capture the
numbers that matter to you. This often means using
an analytics package or writing custom code.

Metrics Plan
•Set a goal
•Set a time span
collect the data
20 min
How will you track
progress?
% of users
who share
a task 3+
times a day,
per week
- timestamp each
of users x 100 = %
30%
8 1 2015
5%
4 19 2015

Pass your work
1-minute “quiet read”
5 minutes of questions
to conﬁrm clarity and
then feedback.
Peers: Offer help and
advice.
Share as a pair
15 min
(total)
Metrics Plan

Tracking over
time

Track over time

Pull it all together.
Tell the story.

This is [product name] .
* Minimum Viable Product
10 minMetrics Gallery
The metric
& plan:
The value:
The idea...

Geek out & learn tons more

Kate Rutter
@katerutter
http://intelleto.com
Thanks!

UX Days Tokyo | Measuring What Matters | @katerutter | April 2015
Handouts
a UX Approach to Metrics
Measuring what Matters

UX Days Tokyo | Measuring What Matters | @katerutter | April 2015
Features
Users
Uses
Product
Needs
Company
Purpose
sketches,
prototypes,
pixels
UI
UX Stack

Product Snapshot
people
problem solution
have
addresses
use
Product name date
/ /

With _________________ , _________________ can...
Key Uses 6-up sketches
date
/ /

With
can...
!
Key Use date
/ /

6
a great metric...
...makes you look at all the other metrics
and say “none of those other numbers
matter if we donʼt get this right ﬁrst.”
a good metric...
...measures the usage of the product by
a person. The usage should be speciﬁc to
features that deliver value to the user.
Making metrics actionable
vanity good betterunhelpful
total number
of registered
users
% of users
who share a
task 3+
times a day,
per week
sign-ups
awesome
% of users
who sign in
3+ times a
day,
per week
% of new
users per
week
Does the metric begin
with a number?
• “percent of...” [ % ]
• “average number of...” [avg ]
• “number of...” [ # ]
Is there a time basis?
Is it comparative?
• “per week” [ ___/wk ]
• “per month” [ ___/mo]
Is there an object basis?
* not required, but super-helpful
• “per user” [ ___/user]
• “per user per week” (fancy!)
1
Metrics Check
2 3

metric
by
Goal
/ /
Notes
Metrics Dashboard date
/ /
To assess progress, we will measure these metrics:
metric
by
Goal
/ /
Notes
metric
by
Goal
/ /
Notes

by
Goal
/ /
Key Metric plan date
/ /
To assess progress, we
will measure this metric:
Measured
Baseline
Daily Weekly Monthly
Hereʼs how weʼll collect the data:
metric

Measuring What Matters: A UX Approach to Metrics :: UX Days Tokyo [April 2015]

More Related Content

What's hot (20)

Similar to Measuring What Matters: A UX Approach to Metrics :: UX Days Tokyo [April 2015] (20)

More from Kate Rutter (20)

Recently uploaded (20)

Measuring What Matters: A UX Approach to Metrics :: UX Days Tokyo [April 2015]