SlideShare a Scribd company logo

Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015

Download as PPTX, PDF
Craig Sullivan, profile picture
Craig Sullivan

The document discusses strategies for successful split testing in analytics, emphasizing the importance of proper setup, research, and prioritization to achieve better ROI. It highlights common pitfalls such as bad analytics setups, poor testing locations, and ego-driven testing, advocating for data-driven decision-making and the generation of actionable insights. The guide culminates in a call to foster agile, collaborative environments to enhance testing and optimization outcomes.

Internet
1 of 86
Downloaded 26 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
Surviving the Hype Cycle Shortcuts to Split Testing Success @OptimiseOrDie, Yo Helsinki!
@OptimiseOrDi e • Split Testing, Analytics, UX • Lean, Agile, Growth Optimisation • 100M+ visitors tested • 200+ sites, 26 languages • Starting Testing? Bad ROI? • Want to scale volume & quality? • Get in touch! @OptimiseOrDiecolinski.tumblr.com
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
@OptimiseOrDie The Gartner Hype Cycle ™
1 Tool Installed 2 Stupid testing 3 4 Peak of Stupidity 5 ROI questioned 6 Statistics debunked 7 Faith crisis 8 The Trough of Testing Scaled up Stupidity 9 Where, How, Why 10 Data science 11 Testing to learn 12 Innovation Testing @OptimiseOrDie
#fai l @OptimiseOrDie
@OptimiseOrDie 26.6M
@OptimiseOrDie28.4M
Oppan Gangnam Style! @OptimiseOrDie6.9M
You been naughty again?
1. Get Analytics Health Checked 2. Test in the right place 3. Understand Cross Device 4. Do your Research 5. Prioritise your testing @OptimiseOrDie 6. Perform Pre Flight Checks 7. Know how long to test 8. Have a good reason to test 9. Learn from your tests 10. Burn down the silos 10 Shortcuts to Testing Success
@OptimiseOrDie 1. Your Analytics Setup is Broken
• Nearly 100 Sites in 3 years • 95% were broken, often badly • Trust in data was missing • Management made bad calls • Nobody checked the tills • Calibrate from the basics up! @OptimiseOrDie • What sales do we capture? • What categories? • What about refunds, lunch money, gift certificates? • How do we monitor fraud? • Do we check it adds up? • Where does this data go? 1. What about MY clients?
@OptimiseOrDie 1. Bulls***t flows upwards! Cool BS Dashboard BS metrics BS Collection BS metrics BS Collection BS metrics BS Collection BS metrics BS Collection BS reports BS Reports TILLS DEPT STORE DIVISIO N
@OptimiseOrDie • Review takes 1-3 days • Prioritise the issues • Fix directly with developers • Integrate with the Testing Tool1 Get an Analytics Health Check
@OptimiseOrDie 2. You Test in the Wrong Place
2. Let’s do Random Testing Let’s try the homepage I’ve got targets to hit! I hate this job Let’s test button colours! Has lots of opinions but no data Spends too much time on Twitter Driven by Ego and Competitors Wishes he cared about testing
@OptimiseOrDie “STOP copying your competitors They may not know what the f*** they are doing either” Peep Laja, ConversionXL 1. Let’s do Random Testing
Best Practice Testing? • Your customers are not the same • Your site is not the same • Your advertising and traffic are not the same • Your UX is not the same • Your X-Device Mix is not the same • You have no idea of the data • Use them to inform or suggest approaches • Use them for ideas • Do not use them as a playbook • It will make you very unhappy @OptimiseOrDie
@OptimiseOrDie 2. Modelling - Intent All traffic HearingSight StoreOther Step 1 Step 2 Step 3 Goal Page PageHearing
@OptimiseOrDie 2. Modelling – Multiple Endings All traffic InfluenceIntent Influence Step 1 Step 2 Step 3 Goal Page Page Entry Page 1 2 3 4 1 2 3 4 1 2 3 4
Bounce Search or Category Product Page Add to basket View basket Checkout @OptimiseOrDie 2. Modelling – Ring Model
2. Modelling – Horizontal Funnels
@OptimiseOrDie • Do some Analytics modelling • Understand the shedding of layers • Narrow your focus and scope • Bank better gains earlier in time 2 Test in the Right Places
@OptimiseOrDie 3. Responsive solves everything, right?
@OptimiseOrDie
@OptimiseOrDie Vs.
@OptimiseOrDie
@OptimiseOrDie
@OptimiseOrDie
@OptimiseOrDie
@OptimiseOrDie 1. Motorola Hardware Menu Button 2. MS Word Bullet Button 3. Android Holo Composition Icon 4. Android Context Action Bar Overflow (top right on Android devices)
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
@OptimiseOrDie Increase in revenue of > $200,000 per annum! bit.ly/hamburgertest
BURGER SHISH DONER
@OptimiseOrDie • Do you really know your mix? • Most people undercount Android! • What iPhone models visit? • How big is tablet traffic? • What screen sizes do they have? • Find out BEFORE you design tests • Check BEFORE you launch tests • Use Google Analytics to find out • 3 reports to rule them all https://www.google.com/analytics/web/template?uid=lpVf8LveSqyd3mdsHjdfzQ https://www.google.com/analytics/web/template?uid=fmUzp_gzRIy7LnvZJjCDOQ https://www.google.com/analytics/web/template?uid=y7sYIXDhQrmswHAiNo8iLA 3. Our customers use iPhones, right?
@OptimiseOrDie 3. What iPhone Models do we see? Screen Resolution 320 x 480 = iPhone 4/4S 320 x 568 = iPhone 5/5S 375 x 667 = iPhone 6 414 x 736 = iPhone 6+ https://www.google.com/analytics/web/template?uid=lpVf8LveSqyd3mdsHjdfzQ https://www.google.com/analytics/web/template?uid=fmUzp_gzRIy7LnvZJjCDOQ https://www.google.com/analytics/web/template?uid=y7sYIXDhQrmswHAiNo8iLA
@OptimiseOrDie • Desktop Browsers & versions • Tablet Models • Mobile Device Models • Screen Resolutions3 Figure Out the Device Mix for Testing
Is there anything holding you from doing conversion research? 1. Time 2. Client/Company Buy-In 3. Budget 4. Don’t know where to start 4. You don’t do any Research before testing?
40
@OptimiseOrDie 4. If you have 4 hours PLUS • Snap interviews (Sales, Customer Services, Tech Support) • Run a quick poll or survey (See my tools slides) Less Bullshit!
Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015
@OptimiseOrDie 4. 1 Hour Page Analytics Influence Pages Entry Points Landing Pages Device Mix Customer Mix Traffic Mix Flow Intent Marketing -> Site flow Page or Process Next Steps Abandonment Exits Mix of abandonment Flow
@OptimiseOrDie 4. 1 Hour Landing Page Analytics • How old are the visitors? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What are the key metrics like (e.g. bounce rate, conversion)? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What is the goal or ecommerce conversion through this page? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What channel traffic comes to the page? https://www.google.com/analytics/web/template?uid=Kjb9q8M4QN-fsPe8dOGaig • What is the mix of tablet / mobile / desktop to the page? https://www.google.com/analytics/web/template?uid=wLMUWs8eTIa3_mmQHOtPkw • What are the resolutions of devices? https://www.google.com/analytics/web/template?uid=wLMUWs8eTIa3_mmQHOtPkw • How slow are the landing pages? https://www.google.com/analytics/web/template?uid=AavFsgMoRkucYYKnxlB76Q • What are the pages right after the landing page? (Use a landing page report and choose the ‘Entrance Paths’ to show next pages.) • What is the flow like from this page? (Use the Behaviour Flow Report) • What does it look like on the top devices? (Use real devices + Appthwack.com, Crossbrowsertesting.com or Deviceanywhere.com)
Ad SERP Banner Email Campaign Aff Referrer Landing Page Template Goal Reached Interaction Step or Layer PPC Organic Display Email Social Desktop - Mobile - Tablet
@OptimiseOrDie 4. If you have 2 hours • Form Analytics data • Scroll or Click Maps • Session Recording Videos (Hotjar, Decibel Insight, Yandex) • Make a horizontal funnel from the landing page • Check the: – Marketing Creatives / SERP fully – Look at Landing page ZERO!
@OptimiseOrDie 4. If you have 4 hours • Set up a poll or survey (See my tools slides) • Set up an exit (bail) survey • Friends, Family, New Employee user testing • Guerrilla user testing • Snap Interviews – 5-10 minutes: • Customer Services, Sales team (if applicable) then Customers • 5 Second Test • Article is here : bit.ly/conversiondeadline
@OptimiseOrDie • Lean Analytics • UX Research • Interviewing • Surveys and Polls4 No Excuses – Do Your Research
@OptimiseOrDie 5. You don’t Prioritise your Tests Scoring can be cost, time to market, resource, risk, political complexity • Cost 1-10 Higher is cheaper • Time 1-10 Higher is shorter • Opportunity 1-10 Higher is greater Score = Cost * Time * Opportunity • For financial institutions, risk should be a factor • Want to build your own? – ask me!
@OptimiseOrDie 5. Opportunity vs. Cost 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 Cheap (high is better) Opportunity MONEY!
@OptimiseOrDie 5. Make a Money Model Test Description Metric 2% lift 5% lift 10% lift Estimate Product page Simplification Basket adds 200,000 500,000 1,000,000 500,000 Register new Improve onboarding New register funnel ratio 25,000 62,500 125,000 250,000 IE8 bugs in cart Fix key broken stuff IE8 Conversion 80,000 200,000 400,000 200,000 Category list page Get product higher User Category -> Product 500,000 1,250,000 2,500,000 1,250,000 Payment Page New card handling User Payment -> Thank you 60,000 150,000 300,000 300,000
@OptimiseOrDie • Score all Test Targets • Use Cost vs. Opportunity minimum • Check it works! • Make a Money Model 5 Prioritise your Testing Targets
@OptimiseOrDie 6. You Don’t Test Before Launch • Dirty secret of AB testing? • People break their tests all the time! • Most people don’t notice Why? • Because developers can break them very easily • What if your AB test was broken on iPhones? • If you didn’t know, would your results be valid? • About 40% of my tests fail basic QA
Browser Checks www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.saucelabs.com Mobile & Tablet www.appthwack.com www.deviceanywhere.com www.opendevicelab.com Article & Info bit.ly/devicetesting 6. Here is my £80M testing rig!
@OptimiseOrDie 6. Always Use Real Devices
@OptimiseOrDie • Check Every Test Works • Browser and Devices • Check Analytics records correctly 6 Perform Pre Flight Checks
@OptimiseOrDie 7. You Stop When you Hit 95% Confidence
The 95% Stopping Problem @OptimiseOrDie • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable and moves around • Nearly all my tests reach significance before they are actually ready • You can hit 95% early in a test (18 minutes!) • If you stop, it could be a false result • Read this Nature article : bit.ly/1dwk0if • Optimizely and VWO have updated their tools • This 95% thingy – must be LAST on your stop list
The 95% Stopping Problem Scenario 1 Scenario 2 Scenario 3 Scenario 4 After 200 observations Insignificant Insignificant Significant! Significant! After 500 observations Insignificant Significant! Insignificant Significant! End of experiment Insignificant Significant! Insignificant Significant! “You should know that stopping a test once it’s significant is deadly sin number 1 in A/B testing land. 77% of A/A tests (testing the same thing as A and B) will reach significance at a certain point.” Ton Wesseling, Online Dialogue
The 95% Stopping Problem @OptimiseOrDie “Statistical Significance does not equal Validity” http://bit.ly/1wMfmY2 “Why every Internet Marketer should be a Statistician” http://bit.ly/1wMfs1G “Understanding the Cycles in your site” http://mklnd.com/1pGSOUP
• TWO BUSINESS CYCLES minimum (week/month) • 1 PURCHASE CYCLE minimum (or most of one) • 250 CONVERSIONS minimum per creative • 350, 500, more if creative response is similar • FULL WEEKS/CYCLES never part of one • KNOW what marketing, competitors and cycles are doing • RUN a test length calculator - bit.ly/XqCxuu • SET your test run time , RUN IT, STOP IT, ANALYSE IT • ONLY RUN LONGER if you need more data • DON’T RUN LONGER just because the test isn’t giving the result you want! @OptimiseOrDie 7. Know How Long to Test for…
@OptimiseOrDie • Most critical mistake • Use a test calculator • Full business cycles, 2 minimum • Don’t waste time hoping7 Know How Long to Test for
Insight - Inputs #FAIL Competitor copying Guessing Dice rolling An article the CEO read Competitor change Panic Ego Opinion Cherished notions Marketing whims Cosmic rays Not ‘on brand’ enough IT inflexibility Internal company needs Some dumbass consultant Shiny feature blindness Knee jerk reactons @OptimiseOrDie 8. So you think you have a Hypothesis?
Insight - Inputs Insight Segmentation Surveys Sales and Call Centre Session Replay Social analytics Customer contact Eye tracking Usability testing Forms analytics Search analytics Voice of Customer Market research A/B and MVT testing Big & unstructured data Web analytics Competitor evalsCustomer services @OptimiseOrDie 8. So you think you have a Hypothesis?
@OptimiseOrDie 1. Because we saw (data/feedback) 2. We expect that (change) will cause (impact) 3. We’ll measure this using (data metric) bit.ly/hyp_kit 8. Use this to deflect stupid testing!
@OptimiseOrDie 1. Because we saw (an angry email from the CEO) 2. We expect that (changing button colours) will cause (the office to cool down for a day) 3. We’ll measure this using (some metric we pluck out of the air – whatever, man) bit.ly/hyp_kit 8. Let’s try a real one
@OptimiseOrDie • Don’t do Ego driven testing • Use the Hypothesis Kit! 8 Get a Proper Hypothesis Going bit.ly/hyp_kit
@OptimiseOrDie 9. Our Testing teaches us Nothing! • Either your research or hypothesis is weak • Work back from the outcome! What if A won – what would that tell us? What if A failed – what would that tell us? • What is the value to the business in finding out the answer? • Is the finding actionable widely and deeply? • Testing isn’t about lifts – it’s about learning
@OptimiseOrDie 9. Our Testing teaches us Nothing! “You are trying to run a bundle of tests, whose expected additional information will give you the highest return.” Matt Gershoff, CEO, Conductrics.com
@OptimiseOrDie • Do your research • Form a solid hypothesis • Work back from the outcomes • Learning useful stuff = huge lifts9 Design Tests for Maximum Learning
@OptimiseOrDie 10. Burn Down the Silos • Non agile, non iterative design • Silos work on product separately • No ‘One Team’ per product/theme • Large teams, unwieldy coordination • Pass the product around • More PMs and BAs than a conference • Endless sucking signoff • AB testing done the same way!
@OptimiseOrDie 10. FT Example • Small teams (6-15) with direct access to publish • Ability to set and get metrics data directly • Tools, Autonomy, Lack of interference • No Project Managers or Business Analysts • Business defines ‘outcomes’ – teams deliver • No long signoff chain • No pesky meddling fools • 18 Month projects over budget?
@OptimiseOrDie 10. FT Example • 100s of releases a day! • MVP approach • Launch as alpha, beta, pilot, phased rollout • Like getting in a shower • Read more at labs.ft.com
@OptimiseOrDie 10. Positive Attributes • Rapid, Iterative, User Centred & Agile Design. No Silos. • Small empowered autonomous teams • Polymaths and Overlap • Toolkit & Analytics investment • Persuasive copywriting & Psychology • Great Testing & Optimisation Tools
@OptimiseOrDie • Agile, Lean, Iterative x silo teams • Ability to get and set metrics • Autonomy, Control, Velocity • Iterative MVP approach • Work on outcomes, not features 10 Burn Down the Silos!
76 “If you think of technology as something that’s spreading like a sort of fractal stain, almost every point on the edge represents an interesting problem.” Paul Graham
77 Time ROI
Rumsfeldian Space• What if we changed our prices? • What if we gave away less for free? • What if we took this away? • What about 3 packages, not 5? • What are these potential futures I can take? • How can I know before I spend money? • McDonalds Hipster Test Store bit.ly/1TiURi7 @OptimiseOrDie
Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy! Innovation Testing @OptimiseOrDie
@OptimiseOrDie
2004 Headspace What I thought I knew in 2004 Reality
2015 Headspace What I KNOW I know Me, on a good day
WE’RE ALL WINGING IT
Guessaholics Anonymous
1 Tool Installed 2 Stupid testing 3 4 Peak of Stupidity 5 ROI questioned 6 Statistics debunked 7 Faith crisis 8 The Trough of Testing Scaled up Stupidity 9 Where, How, Why 10 Data science 11 Testing to learn 12 Innovation Testing @OptimiseOrDie
Thank You! Email me sullivac@gmail.com Slides bit.ly/reaktorOOD Linkedin linkd.in/pvrg14

More Related Content

PPTX
Conversion Research in One Hour
Craig Sullivan
 
PPTX
Cross Device Optimisation - Google Analytics Shortcuts
Craig Sullivan
 
PPTX
20 top AB testing mistakes and how to avoid them
Craig Sullivan
 
PPTX
Interact London - 21 Oct 2015 - Scaling Stupidity
Craig Sullivan
 
PPTX
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
Craig Sullivan
 
PPTX
Surviving the hype cycle Shortcuts to split testing success
Craig Sullivan
 
PPTX
Web Analytics Wednesday - Session Replay Tools are Vital
Craig Sullivan
 
PPTX
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Craig Sullivan
 
Conversion Research in One Hour
Craig Sullivan
 
Cross Device Optimisation - Google Analytics Shortcuts
Craig Sullivan
 
20 top AB testing mistakes and how to avoid them
Craig Sullivan
 
Interact London - 21 Oct 2015 - Scaling Stupidity
Craig Sullivan
 
12 Things to do Before Your Company Dies : Conversion Conference London - Oct...
Craig Sullivan
 
Surviving the hype cycle Shortcuts to split testing success
Craig Sullivan
 
Web Analytics Wednesday - Session Replay Tools are Vital
Craig Sullivan
 
Digital Impact 2014 - Oh Boy These AB tests sure look like Bullshit to me
Craig Sullivan
 

What's hot (20)

PPTX
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
Craig Sullivan
 
PPTX
Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Craig Sullivan
 
PPTX
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
Craig Sullivan
 
PPTX
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Craig Sullivan
 
PPTX
Product design is Poo - And how to fix it!
Craig Sullivan
 
PPTX
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Craig Sullivan
 
PPTX
Myths, Lies and Illusions of AB and Split Testing
Craig Sullivan
 
PPTX
20 Ways to Shaft your Split Tesring : Conversion Conference
Craig Sullivan
 
PPTX
Product Design is Poo - And we're all going to die
Craig Sullivan
 
PPTX
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
Craig Sullivan
 
PPTX
Why do my AB tests suck? measurecamp
Craig Sullivan
 
PDF
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Marketing Festival
 
PDF
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
PRWD
 
PPTX
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Craig Sullivan
 
PPTX
eMetrics London - The AB Testing Hype Cycle
Craig Sullivan
 
PPTX
Natural born conversion killers - Conversion Jam
Craig Sullivan
 
PDF
Onboard like a juggernaut - Elite camp 2015
Conversionista
 
PPTX
John lewis lego craig sullivan 1335
Charlie Lines
 
PPTX
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
Craig Sullivan
 
PPTX
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
Zack Notes
 
#Measurefest : 20 Simple Ways to Fuck Up your AB tests
Craig Sullivan
 
Mobile presentation - Sydney Online Retailer - 26 Sep 2011
Craig Sullivan
 
The Neuromarketing Toolkit - Chinwag Psych - 4 Feb 2014
Craig Sullivan
 
Brighton CRO Meetup #1 - Oh Boy These AB tests Sure Look Like Bullshit to Me
Craig Sullivan
 
Product design is Poo - And how to fix it!
Craig Sullivan
 
Myths and Illusions of Cross Device Testing - Elite Camp June 2015
Craig Sullivan
 
Myths, Lies and Illusions of AB and Split Testing
Craig Sullivan
 
20 Ways to Shaft your Split Tesring : Conversion Conference
Craig Sullivan
 
Product Design is Poo - And we're all going to die
Craig Sullivan
 
#Measurecamp : 18 Simple Ways to F*** up Your AB Testing
Craig Sullivan
 
Why do my AB tests suck? measurecamp
Craig Sullivan
 
Craig Sullivan - Oh Boy! These A/B tests look like total bullshit! MKTFEST 2014
Marketing Festival
 
Why Does My Conversion Rate Suck? Craig Sullivan, Senior Optimisation Consult...
PRWD
 
Condensed testing syrup - @OptimiseorDie @sydney sep 2011 - 4 years of testin...
Craig Sullivan
 
eMetrics London - The AB Testing Hype Cycle
Craig Sullivan
 
Natural born conversion killers - Conversion Jam
Craig Sullivan
 
Onboard like a juggernaut - Elite camp 2015
Conversionista
 
John lewis lego craig sullivan 1335
Charlie Lines
 
UXPA UK - Toolkits and Tips for Blending UX, Analytics and CRO
Craig Sullivan
 
SAMPLE SIZE – The indispensable A/B test calculation that you’re not making
Zack Notes
 
Ad

Viewers also liked (9)

PDF
Conversion rate optimisation 101 @ LS Yorkshire Leeds Feb 2014
Manuel Da Costa
 
PDF
Introduction to conversion rate optimization (cro) for ecommerce
In Marketing We Trust
 
PPTX
Conversion Rate Optimization 101 - Converting Visitors In To Buyers
Kothapally Arun
 
KEY
Designing for Bears
Lucy Spence
 
PDF
Conversion Rate Optimization 101 - Kick-Start Your Growth Engine
Kissmetrics on SlideShare
 
PDF
Conversion Hotel 2016 - Anna Dahlström
Webanalisten .nl
 
PDF
Conversion Hotel 2016 - John Ekman
Webanalisten .nl
 
PDF
Conversion Hotel 2016 - Jeffrey Eisenberg
Webanalisten .nl
 
PDF
Nathalie Nahai - 15 psych tips to help you sell more online
Nathalie Nahai
 
Conversion rate optimisation 101 @ LS Yorkshire Leeds Feb 2014
Manuel Da Costa
 
Introduction to conversion rate optimization (cro) for ecommerce
In Marketing We Trust
 
Conversion Rate Optimization 101 - Converting Visitors In To Buyers
Kothapally Arun
 
Designing for Bears
Lucy Spence
 
Conversion Rate Optimization 101 - Kick-Start Your Growth Engine
Kissmetrics on SlideShare
 
Conversion Hotel 2016 - Anna Dahlström
Webanalisten .nl
 
Conversion Hotel 2016 - John Ekman
Webanalisten .nl
 
Conversion Hotel 2016 - Jeffrey Eisenberg
Webanalisten .nl
 
Nathalie Nahai - 15 psych tips to help you sell more online
Nathalie Nahai
 
Ad

Similar to Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015 (20)

PPTX
Toolkits and tips for UX analytics CRO by Craig Sullivan
UXPA UK
 
PPTX
Toolkits and tips of the conversion pros v 1.6
Craig Sullivan
 
PPTX
Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...
Craig Sullivan
 
PPTX
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Craig Sullivan
 
PPTX
Google Analytics Powerups and Smartcuts
Charles Meaden
 
PDF
Conversion Hotel 2014: Craig Sullivan (UK) keynote
Webanalisten .nl
 
PPTX
Measure camp tools of the cro rabble
Craig Sullivan
 
PDF
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
Northern User Experience
 
PPTX
Elite Camp 2013 - Estonia
Craig Sullivan
 
PDF
Research and Discovery Tools for Experimentation - 17 Apr 2024 - v 2.3 (1).pdf
VWO
 
PPTX
Conversionista : Conversion manager course - Stockholm 20 march 2013
Craig Sullivan
 
PPT
Data Driven Website Decisions for SEO and CRO
Amanda King
 
PDF
Is Growth Hacking Bullshit?
Matt Lerner
 
PDF
It Worked for Ustream
Balázs Kereskényi
 
PDF
Always Be Testing - Learn from Every A/B Test (Hiten Shah)
Future Insights
 
PPTX
12 reasons your site sucks - InvestNI
Craig Sullivan
 
PPTX
Conversion Optimization Willa Fogarty
Willa Fogarty
 
PPTX
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
Invest Northern Ireland
 
PPTX
Growth hacking and Data analysis
Amirhossein Seghatoleslami
 
PDF
Growing Your Business Through Experimentation
Hiten Shah
 
Toolkits and tips for UX analytics CRO by Craig Sullivan
UXPA UK
 
Toolkits and tips of the conversion pros v 1.6
Craig Sullivan
 
Why does my Mobile Conversion rate suck? 19 Sep 2013 @ Conversion Thursday #...
Craig Sullivan
 
Confessions of an uber optimiser conversion summit - craig sullivan - v 1.9
Craig Sullivan
 
Google Analytics Powerups and Smartcuts
Charles Meaden
 
Conversion Hotel 2014: Craig Sullivan (UK) keynote
Webanalisten .nl
 
Measure camp tools of the cro rabble
Craig Sullivan
 
AB Testing and UX - a love story with numbers and people (by Craig Sullivan a...
Northern User Experience
 
Elite Camp 2013 - Estonia
Craig Sullivan
 
Research and Discovery Tools for Experimentation - 17 Apr 2024 - v 2.3 (1).pdf
VWO
 
Conversionista : Conversion manager course - Stockholm 20 march 2013
Craig Sullivan
 
Data Driven Website Decisions for SEO and CRO
Amanda King
 
Is Growth Hacking Bullshit?
Matt Lerner
 
It Worked for Ustream
Balázs Kereskényi
 
Always Be Testing - Learn from Every A/B Test (Hiten Shah)
Future Insights
 
12 reasons your site sucks - InvestNI
Craig Sullivan
 
Conversion Optimization Willa Fogarty
Willa Fogarty
 
Driving Online Sales - Craig Sullivan, The future of the online marketplace 2...
Invest Northern Ireland
 
Growth hacking and Data analysis
Amirhossein Seghatoleslami
 
Growing Your Business Through Experimentation
Hiten Shah
 

Recently uploaded (20)

PDF
Exploring VPS Hosting Trends for SMBs in 2025
Liquid Web
 
PDF
SlidesGDGoCxRAIS about Google Dialogflow and NotebookLM.pdf
minhtrietgect
 
PDF
Understand the Gitlab_presentation_task.pdf
ruhinisiyara
 
PPTX
artificialintelligenceai1-copy-210604123353.pptx
b45r14
 
PPTX
Mathew Digital SEO Checklist Guidlines 2025
Mathew Digital
 
PPTX
module 1-Part 1.pptxdddddddddddddddddddddddddddddddddddd
KevinSeelu
 
PPT
415456121-Jiwratrwecdtwfdsfwgdwedvwe dbwsdjsadca-EVN.ppt
woldemariamworku2
 
PPTX
newyork.pptxirantrafgshenepalchinachinane
PrashantKoirala12
 
PDF
The Evolution of Traditional to New Media .pdf
NIKKODENSMIRO
 
PPTX
Cyber Hygine IN organizations in MSME or
paramkiet
 
PDF
Smart Home Technology for Health Monitoring (www.kiu.ac.ug)
publication11
 
PPTX
Reading as a good Form of Recreation
Raj Kumble
 
PPTX
Database Information System - Management Information System
faizhossain3
 
PPTX
Internet Safety for Seniors presentation
mwnorman1
 
PDF
Introduction to the IoT system, how the IoT system works
TinBinh
 
PPTX
t_and_OpenAI_Combined_two_pressentations
wuvjwfnaadbnzelauy
 
PDF
📍 LABUAN4D EXCLUSIVE SERVER STAR GAMING ASIA NO.1 TERPOPULER DI INDONESIA ! 🌟
alexiskandar061
 
PPTX
IPCNA VIRTUAL CLASSES INTERMEDIATE 6 PROJECT.pptx
AldoCharaRojas
 
PDF
mera desh ae watn.(a source of motivation and patriotism to the youth of the ...
DtMalhonSharma
 
PDF
Alethe Consulting Corporate Profile and Solution Aproach
debashisrakshit2025
 
Exploring VPS Hosting Trends for SMBs in 2025
Liquid Web
 
SlidesGDGoCxRAIS about Google Dialogflow and NotebookLM.pdf
minhtrietgect
 
Understand the Gitlab_presentation_task.pdf
ruhinisiyara
 
artificialintelligenceai1-copy-210604123353.pptx
b45r14
 
Mathew Digital SEO Checklist Guidlines 2025
Mathew Digital
 
module 1-Part 1.pptxdddddddddddddddddddddddddddddddddddd
KevinSeelu
 
415456121-Jiwratrwecdtwfdsfwgdwedvwe dbwsdjsadca-EVN.ppt
woldemariamworku2
 
newyork.pptxirantrafgshenepalchinachinane
PrashantKoirala12
 
The Evolution of Traditional to New Media .pdf
NIKKODENSMIRO
 
Cyber Hygine IN organizations in MSME or
paramkiet
 
Smart Home Technology for Health Monitoring (www.kiu.ac.ug)
publication11
 
Reading as a good Form of Recreation
Raj Kumble
 
Database Information System - Management Information System
faizhossain3
 
Internet Safety for Seniors presentation
mwnorman1
 
Introduction to the IoT system, how the IoT system works
TinBinh
 
t_and_OpenAI_Combined_two_pressentations
wuvjwfnaadbnzelauy
 
📍 LABUAN4D EXCLUSIVE SERVER STAR GAMING ASIA NO.1 TERPOPULER DI INDONESIA ! 🌟
alexiskandar061
 
IPCNA VIRTUAL CLASSES INTERMEDIATE 6 PROJECT.pptx
AldoCharaRojas
 
mera desh ae watn.(a source of motivation and patriotism to the youth of the ...
DtMalhonSharma
 
Alethe Consulting Corporate Profile and Solution Aproach
debashisrakshit2025
 

Surviving the AB Testing Hype Cycle - Reaktor Breakpoint 2015

  • 1. Surviving the Hype Cycle Shortcuts to Split Testing Success @OptimiseOrDie, Yo Helsinki!
  • 2. @OptimiseOrDi e • Split Testing, Analytics, UX • Lean, Agile, Growth Optimisation • 100M+ visitors tested • 200+ sites, 26 languages • Starting Testing? Bad ROI? • Want to scale volume & quality? • Get in touch! @OptimiseOrDiecolinski.tumblr.com
  • 5. 1 Tool Installed 2 Stupid testing 3 4 Peak of Stupidity 5 ROI questioned 6 Statistics debunked 7 Faith crisis 8 The Trough of Testing Scaled up Stupidity 9 Where, How, Why 10 Data science 11 Testing to learn 12 Innovation Testing @OptimiseOrDie
  • 11. 1. Get Analytics Health Checked 2. Test in the right place 3. Understand Cross Device 4. Do your Research 5. Prioritise your testing @OptimiseOrDie 6. Perform Pre Flight Checks 7. Know how long to test 8. Have a good reason to test 9. Learn from your tests 10. Burn down the silos 10 Shortcuts to Testing Success
  • 13. • Nearly 100 Sites in 3 years • 95% were broken, often badly • Trust in data was missing • Management made bad calls • Nobody checked the tills • Calibrate from the basics up! @OptimiseOrDie • What sales do we capture? • What categories? • What about refunds, lunch money, gift certificates? • How do we monitor fraud? • Do we check it adds up? • Where does this data go? 1. What about MY clients?
  • 14. @OptimiseOrDie 1. Bulls***t flows upwards! Cool BS Dashboard BS metrics BS Collection BS metrics BS Collection BS metrics BS Collection BS metrics BS Collection BS reports BS Reports TILLS DEPT STORE DIVISIO N
  • 15. @OptimiseOrDie • Review takes 1-3 days • Prioritise the issues • Fix directly with developers • Integrate with the Testing Tool1 Get an Analytics Health Check
  • 16. @OptimiseOrDie 2. You Test in the Wrong Place
  • 17. 2. Let’s do Random Testing Let’s try the homepage I’ve got targets to hit! I hate this job Let’s test button colours! Has lots of opinions but no data Spends too much time on Twitter Driven by Ego and Competitors Wishes he cared about testing
  • 18. @OptimiseOrDie “STOP copying your competitors They may not know what the f*** they are doing either” Peep Laja, ConversionXL 1. Let’s do Random Testing
  • 19. Best Practice Testing? • Your customers are not the same • Your site is not the same • Your advertising and traffic are not the same • Your UX is not the same • Your X-Device Mix is not the same • You have no idea of the data • Use them to inform or suggest approaches • Use them for ideas • Do not use them as a playbook • It will make you very unhappy @OptimiseOrDie
  • 20. @OptimiseOrDie 2. Modelling - Intent All traffic HearingSight StoreOther Step 1 Step 2 Step 3 Goal Page PageHearing
  • 21. @OptimiseOrDie 2. Modelling – Multiple Endings All traffic InfluenceIntent Influence Step 1 Step 2 Step 3 Goal Page Page Entry Page 1 2 3 4 1 2 3 4 1 2 3 4
  • 23. 2. Modelling – Horizontal Funnels
  • 24. @OptimiseOrDie • Do some Analytics modelling • Understand the shedding of layers • Narrow your focus and scope • Bank better gains earlier in time 2 Test in the Right Places
  • 32. @OptimiseOrDie 1. Motorola Hardware Menu Button 2. MS Word Bullet Button 3. Android Holo Composition Icon 4. Android Context Action Bar Overflow (top right on Android devices)
  • 34. @OptimiseOrDie Increase in revenue of > $200,000 per annum! bit.ly/hamburgertest
  • 36. @OptimiseOrDie • Do you really know your mix? • Most people undercount Android! • What iPhone models visit? • How big is tablet traffic? • What screen sizes do they have? • Find out BEFORE you design tests • Check BEFORE you launch tests • Use Google Analytics to find out • 3 reports to rule them all https://www.google.com/analytics/web/template?uid=lpVf8LveSqyd3mdsHjdfzQ https://www.google.com/analytics/web/template?uid=fmUzp_gzRIy7LnvZJjCDOQ https://www.google.com/analytics/web/template?uid=y7sYIXDhQrmswHAiNo8iLA 3. Our customers use iPhones, right?
  • 37. @OptimiseOrDie 3. What iPhone Models do we see? Screen Resolution 320 x 480 = iPhone 4/4S 320 x 568 = iPhone 5/5S 375 x 667 = iPhone 6 414 x 736 = iPhone 6+ https://www.google.com/analytics/web/template?uid=lpVf8LveSqyd3mdsHjdfzQ https://www.google.com/analytics/web/template?uid=fmUzp_gzRIy7LnvZJjCDOQ https://www.google.com/analytics/web/template?uid=y7sYIXDhQrmswHAiNo8iLA
  • 38. @OptimiseOrDie • Desktop Browsers & versions • Tablet Models • Mobile Device Models • Screen Resolutions3 Figure Out the Device Mix for Testing
  • 39. Is there anything holding you from doing conversion research? 1. Time 2. Client/Company Buy-In 3. Budget 4. Don’t know where to start 4. You don’t do any Research before testing?
  • 40. 40
  • 41. @OptimiseOrDie 4. If you have 4 hours PLUS • Snap interviews (Sales, Customer Services, Tech Support) • Run a quick poll or survey (See my tools slides) Less Bullshit!
  • 43. @OptimiseOrDie 4. 1 Hour Page Analytics Influence Pages Entry Points Landing Pages Device Mix Customer Mix Traffic Mix Flow Intent Marketing -> Site flow Page or Process Next Steps Abandonment Exits Mix of abandonment Flow
  • 44. @OptimiseOrDie 4. 1 Hour Landing Page Analytics • How old are the visitors? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What are the key metrics like (e.g. bounce rate, conversion)? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What is the goal or ecommerce conversion through this page? https://www.google.com/analytics/web/template?uid=hab8Ta93SCCffUpjefjtNQ • What channel traffic comes to the page? https://www.google.com/analytics/web/template?uid=Kjb9q8M4QN-fsPe8dOGaig • What is the mix of tablet / mobile / desktop to the page? https://www.google.com/analytics/web/template?uid=wLMUWs8eTIa3_mmQHOtPkw • What are the resolutions of devices? https://www.google.com/analytics/web/template?uid=wLMUWs8eTIa3_mmQHOtPkw • How slow are the landing pages? https://www.google.com/analytics/web/template?uid=AavFsgMoRkucYYKnxlB76Q • What are the pages right after the landing page? (Use a landing page report and choose the ‘Entrance Paths’ to show next pages.) • What is the flow like from this page? (Use the Behaviour Flow Report) • What does it look like on the top devices? (Use real devices + Appthwack.com, Crossbrowsertesting.com or Deviceanywhere.com)
  • 46. @OptimiseOrDie 4. If you have 2 hours • Form Analytics data • Scroll or Click Maps • Session Recording Videos (Hotjar, Decibel Insight, Yandex) • Make a horizontal funnel from the landing page • Check the: – Marketing Creatives / SERP fully – Look at Landing page ZERO!
  • 47. @OptimiseOrDie 4. If you have 4 hours • Set up a poll or survey (See my tools slides) • Set up an exit (bail) survey • Friends, Family, New Employee user testing • Guerrilla user testing • Snap Interviews – 5-10 minutes: • Customer Services, Sales team (if applicable) then Customers • 5 Second Test • Article is here : bit.ly/conversiondeadline
  • 48. @OptimiseOrDie • Lean Analytics • UX Research • Interviewing • Surveys and Polls4 No Excuses – Do Your Research
  • 49. @OptimiseOrDie 5. You don’t Prioritise your Tests Scoring can be cost, time to market, resource, risk, political complexity • Cost 1-10 Higher is cheaper • Time 1-10 Higher is shorter • Opportunity 1-10 Higher is greater Score = Cost * Time * Opportunity • For financial institutions, risk should be a factor • Want to build your own? – ask me!
  • 50. @OptimiseOrDie 5. Opportunity vs. Cost 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 Cheap (high is better) Opportunity MONEY!
  • 51. @OptimiseOrDie 5. Make a Money Model Test Description Metric 2% lift 5% lift 10% lift Estimate Product page Simplification Basket adds 200,000 500,000 1,000,000 500,000 Register new Improve onboarding New register funnel ratio 25,000 62,500 125,000 250,000 IE8 bugs in cart Fix key broken stuff IE8 Conversion 80,000 200,000 400,000 200,000 Category list page Get product higher User Category -> Product 500,000 1,250,000 2,500,000 1,250,000 Payment Page New card handling User Payment -> Thank you 60,000 150,000 300,000 300,000
  • 52. @OptimiseOrDie • Score all Test Targets • Use Cost vs. Opportunity minimum • Check it works! • Make a Money Model 5 Prioritise your Testing Targets
  • 53. @OptimiseOrDie 6. You Don’t Test Before Launch • Dirty secret of AB testing? • People break their tests all the time! • Most people don’t notice Why? • Because developers can break them very easily • What if your AB test was broken on iPhones? • If you didn’t know, would your results be valid? • About 40% of my tests fail basic QA
  • 54. Browser Checks www.crossbrowsertesting.com www.browserstack.com www.spoon.net www.saucelabs.com Mobile & Tablet www.appthwack.com www.deviceanywhere.com www.opendevicelab.com Article & Info bit.ly/devicetesting 6. Here is my £80M testing rig!
  • 56. @OptimiseOrDie • Check Every Test Works • Browser and Devices • Check Analytics records correctly 6 Perform Pre Flight Checks
  • 57. @OptimiseOrDie 7. You Stop When you Hit 95% Confidence
  • 58. The 95% Stopping Problem @OptimiseOrDie • Many people use 95, 99% ‘confidence’ to stop • This value is unreliable and moves around • Nearly all my tests reach significance before they are actually ready • You can hit 95% early in a test (18 minutes!) • If you stop, it could be a false result • Read this Nature article : bit.ly/1dwk0if • Optimizely and VWO have updated their tools • This 95% thingy – must be LAST on your stop list
  • 59. The 95% Stopping Problem Scenario 1 Scenario 2 Scenario 3 Scenario 4 After 200 observations Insignificant Insignificant Significant! Significant! After 500 observations Insignificant Significant! Insignificant Significant! End of experiment Insignificant Significant! Insignificant Significant! “You should know that stopping a test once it’s significant is deadly sin number 1 in A/B testing land. 77% of A/A tests (testing the same thing as A and B) will reach significance at a certain point.” Ton Wesseling, Online Dialogue
  • 60. The 95% Stopping Problem @OptimiseOrDie “Statistical Significance does not equal Validity” http://bit.ly/1wMfmY2 “Why every Internet Marketer should be a Statistician” http://bit.ly/1wMfs1G “Understanding the Cycles in your site” http://mklnd.com/1pGSOUP
  • 61. • TWO BUSINESS CYCLES minimum (week/month) • 1 PURCHASE CYCLE minimum (or most of one) • 250 CONVERSIONS minimum per creative • 350, 500, more if creative response is similar • FULL WEEKS/CYCLES never part of one • KNOW what marketing, competitors and cycles are doing • RUN a test length calculator - bit.ly/XqCxuu • SET your test run time , RUN IT, STOP IT, ANALYSE IT • ONLY RUN LONGER if you need more data • DON’T RUN LONGER just because the test isn’t giving the result you want! @OptimiseOrDie 7. Know How Long to Test for…
  • 62. @OptimiseOrDie • Most critical mistake • Use a test calculator • Full business cycles, 2 minimum • Don’t waste time hoping7 Know How Long to Test for
  • 63. Insight - Inputs #FAIL Competitor copying Guessing Dice rolling An article the CEO read Competitor change Panic Ego Opinion Cherished notions Marketing whims Cosmic rays Not ‘on brand’ enough IT inflexibility Internal company needs Some dumbass consultant Shiny feature blindness Knee jerk reactons @OptimiseOrDie 8. So you think you have a Hypothesis?
  • 64. Insight - Inputs Insight Segmentation Surveys Sales and Call Centre Session Replay Social analytics Customer contact Eye tracking Usability testing Forms analytics Search analytics Voice of Customer Market research A/B and MVT testing Big & unstructured data Web analytics Competitor evalsCustomer services @OptimiseOrDie 8. So you think you have a Hypothesis?
  • 65. @OptimiseOrDie 1. Because we saw (data/feedback) 2. We expect that (change) will cause (impact) 3. We’ll measure this using (data metric) bit.ly/hyp_kit 8. Use this to deflect stupid testing!
  • 66. @OptimiseOrDie 1. Because we saw (an angry email from the CEO) 2. We expect that (changing button colours) will cause (the office to cool down for a day) 3. We’ll measure this using (some metric we pluck out of the air – whatever, man) bit.ly/hyp_kit 8. Let’s try a real one
  • 67. @OptimiseOrDie • Don’t do Ego driven testing • Use the Hypothesis Kit! 8 Get a Proper Hypothesis Going bit.ly/hyp_kit
  • 68. @OptimiseOrDie 9. Our Testing teaches us Nothing! • Either your research or hypothesis is weak • Work back from the outcome! What if A won – what would that tell us? What if A failed – what would that tell us? • What is the value to the business in finding out the answer? • Is the finding actionable widely and deeply? • Testing isn’t about lifts – it’s about learning
  • 69. @OptimiseOrDie 9. Our Testing teaches us Nothing! “You are trying to run a bundle of tests, whose expected additional information will give you the highest return.” Matt Gershoff, CEO, Conductrics.com
  • 70. @OptimiseOrDie • Do your research • Form a solid hypothesis • Work back from the outcomes • Learning useful stuff = huge lifts9 Design Tests for Maximum Learning
  • 71. @OptimiseOrDie 10. Burn Down the Silos • Non agile, non iterative design • Silos work on product separately • No ‘One Team’ per product/theme • Large teams, unwieldy coordination • Pass the product around • More PMs and BAs than a conference • Endless sucking signoff • AB testing done the same way!
  • 72. @OptimiseOrDie 10. FT Example • Small teams (6-15) with direct access to publish • Ability to set and get metrics data directly • Tools, Autonomy, Lack of interference • No Project Managers or Business Analysts • Business defines ‘outcomes’ – teams deliver • No long signoff chain • No pesky meddling fools • 18 Month projects over budget?
  • 73. @OptimiseOrDie 10. FT Example • 100s of releases a day! • MVP approach • Launch as alpha, beta, pilot, phased rollout • Like getting in a shower • Read more at labs.ft.com
  • 74. @OptimiseOrDie 10. Positive Attributes • Rapid, Iterative, User Centred & Agile Design. No Silos. • Small empowered autonomous teams • Polymaths and Overlap • Toolkit & Analytics investment • Persuasive copywriting & Psychology • Great Testing & Optimisation Tools
  • 75. @OptimiseOrDie • Agile, Lean, Iterative x silo teams • Ability to get and set metrics • Autonomy, Control, Velocity • Iterative MVP approach • Work on outcomes, not features 10 Burn Down the Silos!
  • 76. 76 “If you think of technology as something that’s spreading like a sort of fractal stain, almost every point on the edge represents an interesting problem.” Paul Graham
  • 78. Rumsfeldian Space• What if we changed our prices? • What if we gave away less for free? • What if we took this away? • What about 3 packages, not 5? • What are these potential futures I can take? • How can I know before I spend money? • McDonalds Hipster Test Store bit.ly/1TiURi7 @OptimiseOrDie
  • 79. Congratulations! Today you’re the lucky winner of our random awards programme. You get all these extra features for free, on us. Enjoy! Innovation Testing @OptimiseOrDie
  • 81. 2004 Headspace What I thought I knew in 2004 Reality
  • 82. 2015 Headspace What I KNOW I know Me, on a good day
  • 85. 1 Tool Installed 2 Stupid testing 3 4 Peak of Stupidity 5 ROI questioned 6 Statistics debunked 7 Faith crisis 8 The Trough of Testing Scaled up Stupidity 9 Where, How, Why 10 Data science 11 Testing to learn 12 Innovation Testing @OptimiseOrDie
  • 86. Thank You! Email me sullivac@gmail.com Slides bit.ly/reaktorOOD Linkedin linkd.in/pvrg14

Editor's Notes

  • #3: So – a boring intro bit about me – don’t worry – it’ll be over quickly. You can have a look at your phones for a couple of minutes or just watch David Lynch’s finger instead.
  • #4: And here are some of the clients I’ve been working for recently. Dull bit is now officially over. And I’ll now talk about the big problem with split testing – it’s just not working for a lot of companies.
  • #5: I’ve been working on this presentation and thinking about it for over two months now. And this is one of the first graphs I wanted to include, because it represents the hype cycle in AB testing. This is what a lot of companies go through with new technology adoption, so I wanted to show you what the AB testing hype cycle would look like [CLICK]
  • #7: So if this is what ramping up your AB testing feels like at your company, welcome to the club! You are not alone! I know what’s going wrong because I’ve made all these mistakes myself. Today is about giving you shortcuts away from stupidity and getting you from burning cash to productive testing. My pain is your gain!
  • #8: And this crappy AB testing is basically the equivalent of funny cat videos
  • #9: People taking videos of themselves playing video games
  • #10: And like, wow, there are 6.9 million Gangnam Style videos. Just incredible. But hidden in those big numbers, YouTube will always have a tiny percentage of really great stuff, very little good stuff and a long tail of absolute bollocks. And the same is true of split testing - there's some really well run stuff, getting very good results and there's a lot of air guitar going on.
  • #12: And here are the top 10 reasons – there are about 70 odd ways people manage to break their AB testing but these are the most common mistakes, particularly for companies just starting or scaling up. Let’s run through each one – I’ve included a summary slide of each one, so you’ve got a nice handy checklist to take back to the office.
  • #17: So where do you start testing? Where do you focus your efforts? Over here? This bit over here?
  • #26: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #27: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #28: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #29: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #30: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #31: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #32: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #33: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #34: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #35: It has taken me a long time to find out where all the bear traps are hidden. Mainly from screwing up tests and figuring out what was wrong, through lots of testing time. And most companies and teams are stepping on these bear traps without even realising. And they wonder why the test results aren’t replicated in the bank account results. Hah. I have a list now of about 60 ways to easily break, skew, bias or screw up your tests completely. But here are some real biggies to watch for:
  • #79: And now a bit about something I call Rumsfeldian Space – exploring the unknowns. This is vital if you want to make your testing bold enough to get great results.
  • #82: And this was the state of my head in 2004. The inability to understand what you can and can’t be confident about – but nobody wants to admit they’re fucking guessing a lot of the time. And it took me a long time to figure out I didn’t know anything really – it was all assumptions and cherished notions. It was pretty crushing to test my way to this realisation but MUCH I’m happier now.
  • #83: Now I think I know this much - but I might know a wee bit more than I think I do – but I’m erring on the side of caution. That’s because I'm always questioning everything I do through the lens of that consumer insight and testing. Without customers and data driven insights, you can’t shape revenue and delight. They’ll give you the very psychological insights you need to apply levers to influence them, if you only ask questions. Everything else is just a fucking guess. Even with tests, if the only inputs you’ve got are ego and opinion, they’re going to be lousy guesses and you’re wasting your experiments.
  • #84: I once explained to my daughter – you know, when adults like look really in control and making decisions and appearing not to suffer from indecision? Don’t believe it for a minute – we’re just better at winging it cause we’re older. And this is the huge hole that’s gnawing at the hear of many digital operations. The inability to understand what you can and can’t be confident about – but nobody wants to admit they’re guessing a lot of the time.
  • #85: Hopefully now you’ve joined Guessaholics Anonymous from today, you can move on to a data driven and more productive future. Enjoy less wastage in your products, greater efficiency in marketing spend and grow faster than you imagined. All it takes is a bit of love of your customers, your business and some solid ground rules.
  • #87: I hope you enjoyed my talk as much I did writing it. All my details are here including the slides, for you to download. Go forth and Optimise!