Five Steps to Better Metrics

Five Steps to Better Metrics:
How one marketer leveraged web analytics for an annual
revenue increase of $500,000

#webclinic

Join the conversation on Twitter

#webclinic

#webclinic

Today’s team

Dr. Flint McGlaughlin Jon Powell
Managing Director Senior Manager
Research and Strategy

#webclinic

The Challenge
Q: Are there metrics your organization does NOT monitor, only because they are
not set up properly?

4

#webclinic

Background and Test Design

 Experiment ID: REGOnline Homepage Test
Location: MarketingExperiments Research Library
Test Protocol Number: TP1427

Research Notes:
Background: REGOnline is event management software that lets users create
online registration forms and event websites to manage their events.

Goal: To increase number of completed leads on homepage.

Primary research question: Which page will generate the greatest number of
leads?

Approach: A/B multifactor split test

#webclinic

Experiment: Control
Control - Homepage

• Our researchers
hypothesized that we could
increase the appeal
associated with the value
proposition of this offer by
focusing more on the
product and its specific
features and benefits.

#webclinic

Experiment: Treatment
Treatment - Homepage

• Headline was written to focus
more on the product.

• Specific features and benefits
are utilized to express the
value.

• The page emphasizes “Free
Access.”

• Also, ensured that this value
was being communicated in
subsequent steps.

7

#webclinic

Experiment: Side-by-side
Control Treatment

8

#webclinic

Experiment: Results

24.5% Decrease in Conversion
The Treatment generated 24.5% less completed leads

Conversion
Versions Rel. diff
Rate
Control – Two-step homepage 2.3% -

Treatment – Three-step homepage 1.7% -24.5%

 What youthe amount of form fieldsspite of first step,clearer valuestill
reducing
need to understand: In
in the
having a
the control
and

outperformed the treatment.

#webclinic

Experiment #2: Background
Homepage from
Previous Test
• Before we could get a lift, we
needed to learn more about
the prospects coming to this
site.

24.5%
Decrease in Conversion
• We decided to use one of
their SEO pages as a research
window into the cognitive
psychology of the customer’s
motivation.

10

#webclinic

Experiment #2: Background

 Experiment ID: REGonline SEO landing page test

Research Notes:

Background: A technology and media company specializing in online registration and
event management software.

Goal: To increase the amount of leads generated online.

Primary research question: Which online capture process will generate the higher
addressable lead rate?

Approach: A/B multifactor split test

11

#webclinic

Experiment #2: Control
SEO Landing Page

• This landing page was
offering the same
product as the home
page but dealt with a
smaller subset of
visitors who matched
the profile of those
coming to the
homepage.

• Our researchers could
test here without the
negative consequences
of hurting conversion
on the homepage.

12

#webclinic

Experiment #2: Treatment
Treatment SEO Landing Page

• For our first test on this
page, we tested
focusing on how this
product made the
process of creating
registration forms
easier and could cut
the prospects’ time in
half…

• …and yet it still had a
robust functionality.

#webclinic

Experiment #2: Side-by-side

Control Treatment

Which copy language will generate the most leads?

14

#webclinic

Experiment #2: Results

548% Increase in Complete Leads
The new page’s conversion rate increased by 548.46%

Conversion Rate Relative Statistical Level
Design (%) Difference of Confidence

Original Page 0.7% - -

Treatment 4.8% 548% 99%

 What you need to understand: Bythe treatment wasthis product made
creating registration forms easier,
focusing on how
able to increase
step-level clickthrough rate by 1,312%, and completed leads captured
by 548%.
15

#webclinic

Experiment #2: Final Results

Original Homepage New Homepage Test
SEO Page Test

548% Learning Learning 90%

• We were able to take what we learned about the motivations of their
customers from testing on the SEO landing page and apply it to the
homepage, which generated a 90% increase in leads captured.

16

#webclinic

What we discovered

F Key Principles

1. The goal of all customer research is to enable the marketer to
predict customer behavior.

2. Therefore, the primary usefulness of metrics is not in answering
“how many?” but rather in answering, “why so?”

3. Ultimately, metrics enable the marketer to see the cognitive trail
left by the visitor’s mind.

#webclinic

How do we cut through it all?
• The problem is not typically getting sufficient data from you metrics
software. Rather, the challenge is making sense of it.

18

#webclinic

Online Testing Heuristic

Online Testing Heuristic:

u = 2q + t + m + 2v + i ©

u = Utility
q = Research Question
t = Treatment
m = Metric System
v = Validity Factor
i = Interpretation

19

#webclinic

Today, we will walk through a simple 5-step process for
translating raw testing data into predictive power

20

#webclinic

Translating Raw Data to Predictive Power

F Key Steps
1. Establish Visibility – Ensure that your metric platforms are able to track the
four primary types of analytics:

21

#webclinic

STEP 1: Establish Visibility
Types of Analytics – Visual

Page views referrers
search terms
visitor sessions languages
Amount Source
returning visitors organizations
impressions geographic location

Entry pages
Sign-ups Orders
exit pages
browsers Number of page views

Nature Results
Screen resolution
time on page Click trails
Load errors Most requested pages

22

#webclinic


F Key Steps
four primary types of analytics:

Amount – How many instances of a particular action are occurring?

Source – Where are prospects coming from?

Nature – What are prospects experiencing on your site?

Results – What are prospects doing on your site?

23

#webclinic


F Key Steps
four primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results.

2. Determine Objective – Determine the exact research question you are setting
out to answer with your metrics.

24

#webclinic

STEP 2: Determine the Objective
The Research Question

1. Whether you are running a live test or conducting a forensics metrics analysis,
your research and metrics analysis must be grounded in a properly framed
Research Question.

2. A properly framed Research Question is a question of “which” and sets out to
identify an alternative (treatment) that performs better than the control.

Example:
Not this..
What is the best price for product X?
But this…
Which of these three price points is best for product X?

* Depending on the data available, forensics data is often grounded in a research question of “what?” rather than “which”.
25

#webclinic

Audience Exercise

? How would you refine the following three research questions?

1. What is the best headline for my landing page?
2. Why do I have such a high bounce rate on my offer page?
3. How many objectives should I have on my homepage?

26

#webclinic

The Research Question

1. Often, metrics can also Unique visits
be utilized to Flights
40,607,893
determine the most Hotels
32%
effective research 14,185,646

Autos Not all visitors go
questions you should 7,729,403 through each of
these steps

be asking. Activities
9,167,901
60%
Travelers
2. Metrics can be a 73%
12,883,177

Summary
window into key gaps 58%
7,717,122

into your customer Login
5,665,020
76%
theory and ultimately Contact
3,260,292
into the highest 71%
Payment
potential revenue 2,484,236

opportunities for Completion Rate
once process begins
4% 1,766,609
marketing efforts.

27

#webclinic

Example Case Study – Experiment Background

 Experiment ID: (Protected)

Research Notes:
Background: A website that sells retail and wholesale collector items

Goal: To increase conversion rate

Primary research question: Which version of second step in the conversion
funnel will produce the highest conversion rate?

Approach: A/B variable cluster split test that focused on reducing anxiety
through credibility indicators, copy, and re-organization of existing page
elements
28

#webclinic


Fallout Report: New Customers • When we analyzed the metrics, we
realized there were leaks throughout
the checkout process, the credit card
submission page stood out as low
cost opportunity for immediate
return.

• When we analyzed the metrics even
further, we saw that this step also
had the highest lost revenue per cart
(more than double of any other step).

• From this, we hypothesized that
optimizing this step would have the
highest potential return on our
efforts.

29

#webclinic

Example Case Study – Experiment Control

Control What might be causing the
fallout?

• It is unclear why the credit
card is required when
payment method is
different.

• The complexity of the
Purchase Agreement Terms’
causes confusion and
concern.

• There is no indication that
my credit card information
is secure.
30

#webclinic

Example Case Study – Experiment Treatment

Treatment How we addressed the issues:

• Third-party security indicators
have been added.

• Clearer explanation of why a
credit card is required and that
it will not be charged.

• “Satisfaction Guaranteed”
promise is emphasized.

31

#webclinic

Example Case Study – Experiment Results

5% Increase in total conversion
The new credit card page increased conversion by 4.51%

Design Conversion Rate
Control 82.33%
Treatment 86.04%
Relative Difference 4.51%

 What youthis specific step in theWhile it mighttoseem resulted in a projected
choosing
need to understand:
sales funnel test
like a small increase,

$500,000+ increase in revenue per year. This underscores the potential
impact of a properly identified research question.

32

#webclinic


F Key Steps


3. Track and Measure – Track and measure the appropriate metrics that will
provide you with the answer to your determined research question.

33

#webclinic

STEP 3: Track and Measure
Primary and Secondary Metrics

1. Primary “Test” Metrics: The
essential metrics that enable
you to answer the research
question
Primary
Metrics 2. Secondary Metrics: The
additional metrics you can
utilize to help interpret the
Secondary results of your primary metrics
Metrics

34

#webclinic

Primary Metrics – Examples

Example #1:
Research Question: Which headline will generate the most subscriptions?

Primary Metrics: Visits, subscriptions  subscription rate (%)

Example #2:
Research Question: Which PPC ad will generate the most qualified traffic?

Primary Metrics: Ad spend, conversions  cost per acquisition ($)

Example #3:
Research Question: Which page will generate the most Facebook fans?

Primary Metrics: Visitors, clicks on the “Like” button  fans per visitor (%)

35

#webclinic

Secondary Metrics – Examples

Secondary Metric Potential Insights
Are visitors engaged with the content?
Time on page
Are they confused with the process?

What are visitors interested in?
Click tracking Are they confused with the process?

Is there a lack of relevance to visitors?
Bounce rate Are there too many distractions? Is there too
much (or little) information?
What motivates individual visitor types?
Segment-level data Where are the deeper optimization opportunities?

Form event tracking What form fields cause anxiety or confusion?
How much friction will your visitor put up with?

Traffic patterns Who is coming and where are they coming from?
Can we be more relevant to the visitor?

36

#webclinic


 Experiment ID: (Protected)

Research Notes:
Background: A company offering dedicated hosting services

Goal: To increase the number of leads

Primary research question: Which page design will generate the greater
number of leads?

Approach: A/B multi-factor split test (radical redesign)

37

#webclinic

Example Case Study – Experiment Treatments
Control Treatment

Let’s consider both the primary and secondary metrics
utilized for this test…
38

#webclinic

Example Case Study – Experiment Metrics
Control Treatment

Research Question: Which page design will generate the
greater number of leads?

Primary Metrics Primary Metrics
Visits = 31,400* Visits = 30,560*
leads = 628* Leads = 1,764*
CR = 2.0% CR = 5.7%

Answer: The treatment design will generate 188% more
leads.

* Numbers have been anonymized 39

#webclinic

Example Case Study – Experiment Metrics

• In addition to tracking the
primary metrics, the research
analysts installed some
secondary event tracking
metrics.

• On this page, there were six
expandable sections of copy
featuring different elements of
the product value proposition.

• By monitoring the specific clicks
of visitors on this page, we were
better able to understand what
aspect of this product’s value
proposition was most appealing
to the visitor.
40

#webclinic


F Key Steps



4. Monitor Anomalies – Monitor the data for any anomalies that might indicate a
validity threat.

41

#webclinic

STEP 4: Monitor Anomalies
Audience Question

? What wrong with this test data set?

19.00%

17.00%

15.00%
Conversion Rate

13.00%

11.00% Control

9.00% Treatment 3

7.00%

5.00%

3.00%
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10 Day 11

Test Duration

42

#webclinic

#webclinic
campaign

rates, etc.)
Validity Threats

to a specific online

rates, sales, average
• A more subtle clue is a

of response visitors are
temporary spikes in the

having to a specific online
amount of traffic or views

campaign (e.g., conversion
• Monitor for unexplainable

purchase amounts, bounce
noticeable shift in the kind
3.000

0.000
1.000
2.000
4.000

(3.000)
(2.000)
(1.000)

Saturday, October 11, 2008
Sunday, October 12, 2008
Monday, October 13, 2008
Tuesday, October 14, 2008
Wednesday, October 15, 2008
Thursday, October 16, 2008
Friday, October 17, 2008

YES

NO

Saturday, November 01, 2008
Sunday, November 02, 2008
Monday, November 03, 2008
Tuesday, November 04, 2008
Wednesday, November 05, 2008
Standardized Conversion Rate

Thursday, November 06, 2008
NO

Friday, November 07, 2008
Saturday, November 08, 2008
Sunday, November 09, 2008
Monday, November 10, 2008
Tuesday, November 11, 2008
Wednesday, November 12, 2008
Thursday, November 13, 2008
Graphed results of a 4-week email test with an ecommerce retailer:

Normalized
Normalized B
Normalized Traffic
Normalized Traffic B

43

Validity Threats

Anomalies in your metrics can indicate that there may be validity threats in your
tests and data. Be sure to check for the following validity threats should you
encounter any anomaly.
History Effect – when a test variable is affected by an extraneous variable
associated with the passage of time

Instrumentation Effect – when a test variable is affected by a change in the
measurement instrument

Selection Effect – when a test variable is affected by different types of
subjects not being properly distributed among
experimental treatments

For more on validity threats, see our previous Web clinic replay:
“Bad Data: The 3 validity threats that make your tests look conclusive (when they are deeply flawed).”

44

#webclinic


F Key Steps



validity threat.

5. Interpret Data – Interpret the data by moving from “Which?” to “Why?” to
“What?” to “Where?”.

45

#webclinic

STEP 5: Interpret Data
From Customer Behavior to Customer Theory

Which? Why? What?

Customer Behavior Customer Theory

Which headline will Why this headline?
What does my customer
generate a higher want the most?
response?
Why this testimonial?
Which testimonial will What makes my customer
generate the most especially anxious?
response?

Which call to action will Why this call-to-action? What is my customer’s position in
generate a higher the sequence of micro-yeses?
response?

46

#webclinic

Example Case Study

Again, test results are
interpreted and the next
round of testing is started
for this page

201% 2% 29%

Test results are interpreted and Test is again interpreted and
second test was created based on transferrable principles are
the analyst’s observations applied to other offer pages

47

#webclinic

Where else can we apply this data?

• The discoveries and insights about 451%
customer motivation from the
three prior tests were applied to
other landing pages and used to
optimize PPC campaigns.

• The purposeful effort to identify 302%
and selectively apply these
transferrable insights led to
widespread optimization gains .

257% 28% 603%

48

#webclinic

Baltimore Training Week

Save $100 off any workshop
Promo Code: 284-WS-2022

July 30 - August 1

www.meclabs.com/BTW
49

#webclinic

Summary: Putting it all together

F Key Principles

1. The goal of all customer research is to enable the marketer to predict
customer behavior.

2. Therefore, the primary usefulness of metrics is not in answering “how
many?” but rather in answering, “why so?”

3. Ultimately, metrics enable the marketer to see the cognitive trail left by the
visitor’s mind.

50

#webclinic

Summary: Putting it all together

F Key Steps

1. Establish Visibility – Ensure that your metric platforms are able to track the four
primary types of analytics: (1) Amount, (2) Source, (3) Nature, (4) Results.

2. Determine Objective – Determine the exact research question you are setting out to
answer with your metrics.

3. Track and Measure – Track and measure the appropriate metrics that will provide
you with the answer to your determined research question.

validity threat.

5. Interpret Data – Interpret the data by moving from “Which?” to “Why?” to “What?”
to “Where?”.

51

#webclinic

Audience Question

How can I track and integrate social media metrics
? into my web analytics?

-Anne

52

#webclinic

Audience Question

Is Google Analytics "good enough“ to measure
? everything I need?

-Lou

53

#webclinic

Audience Question

What is the best method for calculating
? incremental click costs for low volume keywords?

- Don

54

#webclinic

Audience

How should I interpret bounce rates?
?
- Steve

55

#webclinic

MarketingExperiments.com/subscribe
56

#webclinic

Five Steps to Better Metrics

More Related Content

What's hot (20)

Similar to Five Steps to Better Metrics (20)

More from MarketingExperiments (10)

Recently uploaded (20)

Five Steps to Better Metrics