How to Run Smarter A/B Tests, Faster

How to Run Smarter A/B Tests, Faster
@shanelle_mullin
We’ll cover…
1. How to conduct conversion research to find smarter A/B test ideas.
2. How to prioritize your A/B test ideas in a meaningful way.
3. What a high velocity testing program is and why it works.
4. Common high velocity testing program mistakes to avoid.
5. How to increase your testing velocity responsibly.

“You don’t have generic problems, you
have specific problems.”
~Peep Laja

1. Relevancy
Does the page’s content meet user expectation?
Does the page’s design meet user expectation?
How can it match what they expect / want even more?

2. Clarity
Is the content on this page as clear as possible?
Is the value prop on this page as clear as possible?
How can we make it clearer?

3. Motivation
What’s motivating the visitor to take action?
Is it clear, believable, timely and persuasive?
How can we increase motivation even more?

4. Friction
What is causing doubts, hesitations and uncertainties?
What makes the process difficult?
How can we simplify the process?

5. Distraction
What’s not helping the user take action?
What’s drawing attention?
Is everything that draws attention really necessary?

Cross-Browser Testing
Audience > Technology > Browser & OS
Switch from the default “Data” view to the “Comparison” view.

Cross-Browser Testing
Always compare within the same browser family.

Cross-Device Testing
Audience > Mobile > Devices

Speed Analysis
Page Load Time: Seconds until every element on the
page is done loading.
Page Interactive Time: Seconds until the site is usable.

Speed Analysis
Your site loads in 3 seconds or less.
Your site loads in 4 to 7 seconds.
Your site loads in 7 to 10 seconds (or more).

Speed Analysis
Behavior > Site Speed > Page Timings

Google Analytics Health Check
1. Are you collecting all of the data you need?
2. Can you trust the data you are collecting?
3. Is anything broken or tracking / reporting
incorrectly? Why?

Common Google Analytics Mistakes
1. Pages are missing.
2. Data is different in the shopping cart tool.
3. Cross-domain tracking isn’t working.

Full Health Check Guide
bit.ly/gahealthcheck

Start with the Highest Value
1. High volume, low value pages.
2. Low volume, high value pages.

Finding Broken Links

Auditing Your Funnel
The further down the funnel, the smaller the
necessary impact.

Auditing Your Funnel

Exploring Internal Search

Segmentation Is the Secret
bit.ly/gasegmentation

Click Maps
1. This info is largely available in Google Analytics via
“Navigation Summary”.
2. See when people are clicking on non-links.

Scroll Maps
1. Better prioritize your content and messaging.
2. Drastic colour changes mean a need for visual
cues… likely indicate false bottom.

Session Replays
1. Real users with real money.
2. Where do they pause? Where do they give up?
3. Use this to gather more insight on poor performers.

Form Analytics
1. Which form fields cause the most error messages?
2. Which do people hesitate to fill?
3. Which do people leave empty, even if required?

On-Site Surveys
Exit surveys.
On-page surveys.

On-Site Survey Questions
1. What was the purpose of your visit to our site?
2. Were you able to complete your task?
3. If not, why?
4. Is there anything holding you back?
5. Do you have unanswered questions?
Focus on actionable data, on the friction.

On-Site Survey Options
1 open-ended question.
Y/N and then an explanation post-answer.

On-Site Survey Guide
bit.ly/onsitesurveys

Customer Surveys
1. First time buyers.
2. No previous relationship with you.
3. Try for ~200 responses.

Customer Survey Questions
1. Who are you?
2. What are you using this product for?
3. What problem does it solve for you?
4. What do you like about the product most?
5. Did you consider any alternatives?
6. Why did you choose us over the competition?
7. What’s the one thing that nearly stopped you from buying?
8. What was your biggest concern or fear about us?

User Testing Tasks
1. A specific task.
2. A broad task.
3. Funnel completion.

Constant Ideation
1. Continue conducting conversion research.
2. Involve the entire team.
3. Focus on the entire funnel (acquisition to referral).

“If you come to me with an idea and it’s not
live in 2 weeks, it’s not because it’s a bad
idea. It’s because I have better things to test.”
~Pauline Marol

The Prioritization Problem
1. Not objective enough.
2. Not data-driven enough.
3. Not easily customized.

Benefits of PXL
1. Makes “potential” more objective.
2. Makes “ease” more objective.
3. Helps create a data-driven culture.
4. Easy to customize.

Prioritization Spreadsheet
bit.ly/pxlmodel

Better Management
1. Review KPIs.
2. Review tests launched (and not launched).
3. Review insights gained.
4. Choose tests for the upcoming week.
5. Discuss resources (or lack thereof).

Archive (Or Live to Regret It)
1. Avoid repeating tests by mistake.
2. Easily communicate wins and insights.
3. Improve the quality of future tests.
4. Create an insights > wins culture.

High Velocity Mistakes
1. Without buy-in and a testing culture, you’re
doomed.
2. Calling it too soon and non-representative samples.
3. Letting quality and effectiveness slip.

Testing Capacity
(52 / Avg. Req. Test Duration) x Number of Testable Pages
(52 / 2) x 10 = 260

Measuring Velocity
1. How well are you trending towards achieving your
capacity?
2. Trends are what matters… are you getting closer?
3. Chart it.

@shanelle_mullin
1. Conduct proper conversion research.
2. Come up with ideas constantly.
3. Prioritize with data.
4. Manage the process.
5. Archive your learnings.
6. Achieve your testing capacity.
7. Stop listening to CRO BS you read online.

Resources
Optimization Training Program
bit.ly/cxlinstitute
Original UX Research
bit.ly/cxlresearch

Questions?
Ask any time via email: shanelle@conversionxl.com

How to Run Smarter A/B Tests, Faster

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to How to Run Smarter A/B Tests, Faster (20)

Recently uploaded (20)

How to Run Smarter A/B Tests, Faster