Why A/B testing is probably damaging your business (and what you can do about it)
1 like112 views
This document discusses common problems with A/B testing and provides alternatives. It notes that many companies test pointless metrics that do not impact revenue, have too small sample sizes to be confident in results, and stop tests too soon. Instead, it recommends user testing with 5-15 users to identify usability issues, analyzing site copy, design, and speed, and running tests over multiple business cycles. The key message is that A/B testing should only be used if enough traffic is available and tests should focus on metrics that directly influence revenue.
Why A/B testing is probably damaging your business (and what you can do about it)
- 1. Why A/B testing is probably damaging your business (and what you can do about it)
- 2. We need to have a serious talk. About A/B testing.
- 3. Most people are doing it wrong.
- 4. "visitors spent 67% more time on my website after I changed my headline" "23% more people click the red button" "42% increase in email open rates when I included my brand in the subject line" "adding more images got me 18% more downloads" Reason #1: Pointless tests
- 5. That's all interesting, but: So what if you got more clicks? What difference does it make if someone spends 1 minute or 10 minutes on your site? Why does it matter how many people open your email?
- 6. Only test something that's going to have a direct impact on your revenue.
- 7. Anything else is just a waste of time.
- 8. Before running a test, use this checklist: - What are we testing? - What improvement are we trying to achieve? - What difference will it make to our revenue? - How will we know if it's been a success? - What will we do if we don't find a winner? (don't worry, the majority of tests won't give you a clear winner)
- 9. Reason #2: Not enough people in your sample group
- 10. "People don't think how they feel, they don't say what they think, and they don't do what they say" David Ogilvy People are unpredictable. So the more people you include in your sample, the more confident you can be that your results aren't down to chance.
- 11. How to find out how many you need Use a sample size calculator, like: http://www.evanmiller.org/ab-testing/sample-size.html
- 12. Let's take a closer look at those figures. imagine your current page converts at 20%... and your A/B test gives you 5% improvement... this is the number of people who need to have seen each version. That's 50,510 people... These figures mean you can be 95% sure of your result.
- 13. "What? But... I don't have that many visitors!"
- 14. That's ok... you can still optimise your website. Just not through A/B testing. Otherwise you risk making business decisions based on the wrong data. And you could end up removing what's working on your website, and keeping what isn't.
- 15. So what can you do? Plenty...
- 16. User testing Get real live feedback from real live humans 1) recruit 5–15 users (they will catch 95% of the problems on your site) 2) set some tasks, like: - show them your homepage for 8 seconds and then ask them if they know what the site offers - ask them to complete some tasks (find a category, compare products, make a purchase – that sort of thing) 3) get their feedback (look for common themes or phrases they use, see where they had problems – and fix them)
- 17. Copy analysis checklist - what is this website about? Does the copy explain, in a way that shows a benefit to the reader and gives reasons to stick around? - can I trust it? Are there testimonials, reviews, credit card logos, security logos? - is it clear what I can do here? Am I meant to read, watch a video, make an enquiry, give my email address? Go through every page and make sure each one answers these questions:
- 18. Design analysis checklist Go through every page and make sure each one answers these questions: - What is the most important thing on this page? Make sure the most important stuff (eg headline, image of happy customer, list of benefits) stands out. - Is everything where I expect to see it? The menu at the top, search box top-right, the most important information high up the page, so users don't have to scroll. - Are there any distractions? Do the pictures and layout help the user or does it just look good without guiding them?
- 19. Set up video recording and watch how your visitors interact with your site. Use a tool like: Record your visitors
- 20. - where your users click - what they look at - what is ignored or missed - where users have problems on the page - how far down the page they scroll Use Hotjar to learn: Hotjar clickmap screenshot Hotjar scroll map. Colours get cooler where fewer people scroll
- 21. Test site speed 40% of users leave a site if it takes longer than 3 seconds to load Every second added to page loading speed = 7% drop in conversions If your site is slow, expect to slide down the Google rankings.
- 22. Want to find out if your website is slow? Plug in your URL to tools like: - https://tools.pingdom.com (you can also dig into Google Analytics under Page Speed Insights. This is takes you beyond the averages, and shows specific pages that are slow) - https://gtmetrix.com
- 23. Reason #3 The test is stopped too soon
- 24. Some A/B testing tools will say you have a winner, too early.
- 25. If your repeat visitors arrive and see your test page, they're likely to be curious and explore more than usual. This is dangerous
- 26. That's the 'novelty effect' in action Wait for your visitors to get used to your test version. Then your results will settle down. And you can start unovering the treasure hidden within your data. At the start of an A/B test, results are always erratic.
- 27. - Which seasons or times of the year are traditionally busy?
- 28. If your target audience is people with salaried jobs, they're more likely to be shopping around the end of the month. 'When is payday?'
- 29. Think how people behave on Friday night...
- 31. Before deciding on a winner, work out your business cycle. Then run tests over 2 cycles.
- 32. Know what's going on in your industry and how this might affect your test results: - Black Friday An increase in visits from bargain hunters - Seasonal trends People shopping for Christmas or summer holidays - Competitor goes bust You then get an increase in visits from people who may not have visited your site before - Communication breakdown Another department in your organisation publishes a press release or starts a social media campaign which leads to a sudden increase in visits
- 33. Don't worry if you don't find a winner That's fine. You'll still find out what doesn't have an effect. So you can try something else next time. "I haven't failed. I've just found 10,000 ways that won't work." Thomas Edison The majority of tests don't give a clear winner.
- 34. And relax. Thanks for staying until the end. Here's a summary: - Do A/B tests if you have enough traffic - If you don't, do a 'best practice' analysis - carry out user testing - run tests for 2 business cycles - don't stop the test too early - don't worry if you don't find a winner
- 35. We've just scratched the surface. So I hope you've found something useful among it all. If you have any questions about the slides, or you want me to go more in-depth about any of it, get in touch via LinkedIn: https://uk.linkedin.com/in/stevealphabet