Mastering Analytics for Optimisation Success

@michelejkiss #CH2017
Mastering Analytics
for Optimisation Success
Michele Kiss, Analytics Demystified

About me

Analytics is my jam

So why am I here?

Guide Your Success Metrics

Analytics should help you
define your experiment’s success metric

define your experiment’s success metric

A single success metric
•  checking your eligibility; or
•  eligibility rate; or
•  conversion to purchase
✗

Analytics is one method (of many)

@michelejkiss #CH2017@michelejkiss #CH2017

hypothesis hī- pä-th -s s
a tentative assumption made in
order to draw out and test its
logical or empirical
consequences

I believe that…
{ some idea }

I believe that…
If I am right, then I will…
{ some idea }
{ take some action }

Credit where credit is due
@tgwilson

Hypothesis
Testing Analysis User Testing Survey
Investigative
Pre/Post
A/B/n
Multivariate
Web
Email
Lab
Online

Hypothesis
Testing Analysis User Testing Survey
Investigative
Pre/Post
A/B/n
Multivariate
Web
Email
Lab
Online
✗

Join forces
Join forcesJoin forces

Analysis Can Inform Test Ideas

Reporting /
Analysis
Opportunity
Test runsResults
Further
analysis

{example} Analysis Informs Ideas

Analysis + Survey

“I suppose it is tempting,
if the only tool you have is a
hammer, to treat everything
as if it were a nail."
Abraham Maslow, “The Psychology of Science” (1966)

Help You Prioritise Test Ideas

Prioritising requests

What sells your idea better?
“I believe that …[test idea]”

What sells your idea better?
“I believe that …[test idea]”
vs
“I believe that [test idea]…
and with a 2% increase in conversion, that
would drive a $1MM increase in revenue”

{example} Revenue Forecast
mkiss.me/forecast-impact
•  Use existing data
-  Traffic, conversion, revenue
•  Layer on assumptions
•  Estimate impact

•  Current traffic
•  Conversion to product pages
•  % that view images
•  Current conversion rate
•  Current conversion rate for image-viewers

Add your assumptions!
•  What % of Product Page and/or Image
Viewers will view Video?
•  What % lift in conversion will that drive?
•  Any increase in AOV?
•  Incremental revenue gain?

mkiss.me/forecast-impact

Track Forecast vs. Actual
(Keep people honest!)

Better control the uncontrollable

Statistical significance is not enough
“Another way to think of it is that statistical significance isn’t enough.
Large sites … get thousands of conversions per hour, and can see
significance from their testing efforts quite quickly.
But this doesn’t mean they should only run their tests for a few hours.
Rather, they should run their tests long enough to capture a
representative sample of their users over time.”
- Peter Borden, Sum All

Stationary vs. Non-Stationary Data
Discussion credit: @mgershoff

Problem #1: Seasonality
Artwork © Ryan Kapp Discussion credit: @mgershoff

Understand your seasonality
6.0%
6.5%
7.0%
7.5%
8.0%
8.5%
9.0%
9.5%
10.0%
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 1
Year 2
Year 3
Year 4
Year 5

Problem #2: Drift
Discussion credit: @mgershoff

Analytics Can Lend Credibility

Ever doubted your testing tool?

{example} Homegrown Solution
Are users falling in to multiple test variations?

{example} Homegrown Solution
Deployed on a new brand:

{example} A/A Tests
Control Control
vs

{example} A/A Tests
A vs. A:
60% lift
with
99.94%
confidence

{example} A/A tests
Digging in…
Influx of paid media

{example} New Signup Flow
Step 1 Step 2 Step 3 Signup
Complete
Step 1 Step 11 Signup
Complete

The Problem?
Step 1
Step 1 Step 11 Signup
Complete

The Problem?
Complete
Step 1

Overlapping Test Groups
Conversion

Meh.
I think we’ll just launch it.

Launch vs. Test
Much as we don’t like it…
Some things can’t/won’t be tested.

{example} Pricing Optimization

{example} Site Redesign
Control Redesign
vs
Performed
-3% worse

{example} Site Redesign
Redesign CTA
Animation Animation+CTA
Control
vs

Diving deeper in to results

Don’t forget your success metric!
This isn’t permission to judge your test by
“anything positive you can find”!

{example} Landing page redesign
Control
vs
RedesignCheck
1 Smith Street, San Jose Check
Check
Check
Check
Performed
-8% worse

Control
vs
RedesignCheck
Check
Check
Check
Performed
Similarly

Control
vs
RedesignCheck
Check
Check
Check

Control
vs
RedesignCheck
Check
Check
✔

{example} Skewed test
Control
Variation 1
Variation 2
Variation 3
Variation 4
✔

Control
Variation 1
Variation 2
Variation 3
Variation 4
✔
✗

Not the only time! Months later:
huge spike in traffic to control

{example} Signup Flow Test
Complete
Steps
1-X
Signup
Complete
Step 4 Step 5 Step X
Performed same or
-20% worse

•  Channel
•  Customer Type (Consumer vs. SMB)
•  Device
•  Operating System
•  Browser

spidey-sense

Testing Past Online Conversion

{example} Trial to Paid

{example} Trial Period
Control
~~~~~~~~~~~~
90-DAY TRIAL!
~~~~~~~~~~~~
~~~~~~~~~~~~
~~~~~~~~~~~~
~~~~~~~~~~~~
~~~~~~~~~~~~
~~~~~~~~~~~~
Variation
~~~~~~~~~~~~
30-DAY TRIAL!
~~~~~~~~~~~~
~~~~~~~~~~~~
~~~~~~~~~~~~
~~~~~~~~~~~~
~~~~~~~~~~~~
~~~~~~~~~~~~
No diff in trial starts
+5% paid conv.

{example} eCommerce

Inform Non-Traditional Testing

{example} Content Site
Redesign
PVs/Visit

Web + Ad Analytics allowed us to value
every.single.page.
• How many (and which ads) are on that page
• How well sold the ads were
• What CPM we were charging

Go forth and analyse!
ü  Find opportunities
ü  Estimate the impact
ü  Dig in to your results
ü  Take your testing further
ü  Oh, and befriend your
analysts!

Questions?
@MicheleJKiss
michele@analyticsdemystified.com
join.measure.chat

Michele Kiss
michele@analyticsdemystified.com
@michelejkiss
join.measure.chat

Mastering Analytics for Optimisation Success

More Related Content

What's hot (20)

Similar to Mastering Analytics for Optimisation Success (20)

Recently uploaded (20)

Mastering Analytics for Optimisation Success