Driving Conversions and Revenue With A/B Testing
- 1. Magento Webinar Series: Driving Conversions & Revenue with A/B Testing © 2016 Magento , Inc. Page | 1 o
- 2. © 2016 Magento , Inc. Page | 2 Kevin Eichelberger CEO, Blue Acorn Meet The Speakers Sean Fisher Director of eCommerce , Signature Hardware
- 3. eCommerce A/B Testing A quick p rimer
- 4. What isA/BTesting? #optimizeba An approach to test hypotheses with your customers by measuring their response to different versions of your offer.
- 5. WhyA/BTest? 1. To learn more about your customers 2. To drive measured KPI improvements #optimizeba
- 6. What do we A/BTest? User Interaction & Interface • Workflows • Features • Visual hierarchy • Usability Psychological Factors • Voice • Promotions • Anxiety • Urgency • Buying modality The process of buying something (experience) The reasons to buy something (offer)
- 7. Are you sure about that? How do you know if the features you’ re chang ing actually help or hurt?
- 8. What a deal! Right? #optimizeba
- 9. What a deal! Right? CBA D +12.3% #optimizeba
- 10. Product Reviews Help, Right? #optimizeba
- 11. A +10.5% B Product Reviews Help, Right? #optimizeba
- 12. I’ve got an awesome new page redesign #optimizeba
- 13. I’ve got an awesome new page redesign #optimizeba
- 14. A -7.2% B I’ve got an awesome new page redesign #optimizeba
- 15. Less is more. Or is more less? Or is, wait a minute… #optimizeba
- 16. Nope, more is more. A #optimizeba B +10.5%
- 17. When changing anything on your site DOs • Start with a hypothesis about that change • Use behavioral data as evidence why the change is needed • Gather qualitative data about the proposed change you’re making • A/B test the change itself to validate your hypothesis DON’T’s • Come up with a list of ideas to change on your site based solely on opinions • Expect all of your changes to improve the experience • Invest your resources foolishly–take calculated bets
- 18. Small Devices, Big Results Op timizing the mobile s hop p ing exp erience
- 31. Keep it tidy
- 33. When improving your mobile experience DOs • Gather qualitative data about your mobile experience & identify areas of improvement • Isolate “experience” A/B tests for mobile different than desktop • Analyze A/B test data down to the device type DON’T’s • Go solely by the opinion of your designer on what the best “mobile first” experience is • Think that just by having a responsive site that you’ve handled mobile • Assume modal boxes work effectively in mobile experiences
- 34. Promoting Promotions G etting more out of your p romotions with the rig ht treatment
- 35. Promo code, promo code, where for art thou? #optimizeba
- 36. A +8.9% B Promo code, promo code, where for art thou? #optimizeba
- 37. Promo code, promo code, where for art thou? #optimizeba
- 38. +5.3% BA Promo code, promo code, where for art thou? #optimizeba
- 39. A little promotional oomph #optimizeba
- 40. +2.7% B A A little promotional oomph #optimizeba
- 41. When offering promotions DOs • Test the messaging of your promotions to find what resonates best • Find opportunities to reinforce the promotions throughout the experience • Find the balance of making sure the promotion is seen but not obtrusive DON’T’s • Assume that a homepage promotion or global banner is effective • Think that you can’t get more out of your promotions
- 42. Surprising Results A little inspiration to get you thinking outside the box #optimizeba
- 46. Offline psychology to Online #optimizeba
- 47. +12.2% BA Offline psychology to Online #optimizeba
- 51. When A/BTesting DOs • Have a commitment to testing over time • Use qualitative and quantitative data to determine what to test • Find opportunities for compounding wins • Fundamentally understand the value of testing – learning and wins DONTs • Expect to get huge lifts from your tests • Use these specific test examples and change things on your site • Test random stuff just because you can • Expect a majority of your tests to be “winners”
- 52. Q & A #optimizeba blueacorn.com
Editor's Notes
- #4: Thank you for that great introduction, Emily, and for partnering with us to bring this information to those who were unable to addend the session at Imagine. We’ve updated the content and are glad to have Sean join us as well. What we’re going to share with you today is over a dozen of our most recent A/B tests that we’ve run with various clients. Sean has been gracious enough to share his story from one such client of ours – a merchant’s perspective on A/B Testing All of the test results we’re sharing with you today have been tested in one of our three recommended testing platforms – Monetate, Qubit, or Optimizely, and have all reached statistical significance with tests typically running anywhere from 2 weeks to 2 months in duration.
- #6: SEAN: Can you tell us a little about why you test?
- #7: SEAN: Can you tell us how you determine what to test (or how you develop a testing roadmap)?
- #8: Intro then… SEAN: how have you changed the approach at Signature Hardware when we look your own roadmap of features and enhancements you make to the site?
- #10: This was actually two tests 12.3% lift in revenue 96.3% statistical confidence Ironically, B and C performed worse than A
- #11: Lots of turnover in products, not enough time to generate customer reviews before product was replaced with another Left the site with lots of products without reviews, or with very few Hypothesis was that this created a negative social proof
- #12: 10.5% increase in revenue per visitor 94% statistical confidence Despite what some of the biggest reviews platforms tell you – test for yourself how effective reviews are
- #13: Need Supply new product page layout
- #15: Need Supply new product page layout
- #16: SEAN: Can you walk us through this test and what you were trying to determine here, along with the result? (Resulted in a 10.5% increase in revenue per visitor.) 94% statistical confidence
- #17: 10.5% increase in revenue per visitor 94% statistical confidence
- #18: Behavioral data could come in the form of your own customers, or in the form of published research or studies SEAN: Anything to add?
- #20: Kevin: intro test; SEAN: Can you want to walk us through this test?
- #21: 11.1% increase in conversion rate 6.6% increase in product click-throughs
- #23: 6.2% increase in CR 8.5% increase in product page click-throughs
- #25: 32.2% increase in RPV - 95% stat sig >100% increase in homepage click engagement - 99% stat sig 21.8% increase in conversion rate - 90% stat sig
- #26: Here’s another example of a mobile test. SEAN: Can you walk us through this one?
- #27: 10.5% increase in revenue per visitor
- #29: 6.2% increase in revenue
- #30: AJAX add to cart
- #31: 6.9% increase in mobile RPV Interestingly – this same treatment on desktop performed worse -9.1%
- #33: 11.8% increase in RPV - 79% stat sig 4.7% increase in add to cart rate - 99% stat sig 1.2% decrease in bounce rate - 97% stat sig 11.6% increase in conversion rate - 88% stat sig 15.1% increase in new visitor conversion rate - 81% stat sig
- #34: SEAN: Anything to add here?
- #36: Raise your hand when you see the promo code on this product page
- #37: 8.9% life in RPV
- #38: VIC promo code in product details
- #39: 5.3% increase in conversion rate
- #40: SEAN: Can you talk about this test, where we added a free shipping notification in the header?
- #41: 2.7% increase in RPV
- #42: SEAN: Anything to add?
- #44: BARCO cart page
- #45: BARCO cart page
- #46: 38.4% increase RPV 23.7% increase in conversion rate
- #47: VIC pricing removing the .00
- #48: VIC pricing removing the .00
- #49: 12.2% increase in RPV
- #50: Affirm payment options – for one particular client or across the board
- #51: Offering a payment option through Affirm increase AOV 118% CR increased 7%
- #52: Affirm payment options – for one particular client or across the board
- #54: SEAN? Additional thoughts?
- #55: Sean – I’m going to kick things off with the first question – when we looked at some of your results – we see mostly single-digit gains – these don’t seem like the typical, flashy case studies where someone changes a button color and sees a 300% lift – why do you consider your A/B testing to be successful?