Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.
1 like3,752 views
The document outlines 18 common mistakes in A/B testing and experimentation programs, emphasizing the importance of hypothesis-driven testing and proper processes. Key mistakes include launching without testing, expecting immediate significant results, and failing to conduct quality assurance on experiments. It highlights the need for a systematic approach to testing and continuous improvement to enhance business outcomes and avoid common pitfalls.
Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.
- 1. Learn from the 18+ mistakes that I’ve made while running experimentation programs & how to overcome them. Bad Experiments: The #18 Ways You’re A/B Tests are Going Wrong.
- 2. Martijn Scheijbeler Growth & SEO at Postmates Former Marketing Director at The Next Web martijn.scheijbeler@postmates.com @MartijnSch
- 3. Page
- 4. Page You finally realized: “We want to do more A/B Testing for our - Every Manager Ever
- 5. Page 5 What I’m seeing: Most Experimentation Programs Are Based on Guesswork
- 6. Page 6 @MARTIJNSCH #LAUNCHSCALE 450%900%1,350%1,800% Janaury February March April May June July August September October November December
- 7. Page Or in other words: Given the options, I prefer to learn from success. 7 @MARTIJNSCH #LAUNCHSCALE
- 8. Page Ideas Quality Assurance Design & Engineering Hypothesis Run & Analysis Repeat A/B Testing Repeat this process… 8 @MARTIJNSCH #LAUNCHSCALE
- 9. Page Or in other words: Test. Eat. Sleep. Repeat. 9 @MARTIJNSCH #LAUNCHSCALE
- 10. Page 10 WHAT CAN YOU TEST? You want to know what might improve the bottomline. Ideas Generate a backlog of ideas You want to generate a lot of ideas to make sure that there are always new ways to keep on testing with the current resources that you have. By having a longer backlog you have the opportunity to test and prioritize at the same time. PROCESSES ARE ESSENTIAL Ideas Quality Assurance Design & Engineering Hypothesis Run & Analysis Repeat A/B Testing Repeat this process… @MARTIJNSCH #LAUNCHSCALE
- 11. They just launch, they don’t test. MISTAKE #1 By not testing what you’re launching you’re setting yourself up for failure. MISTAKE #2 “We’re losing money by testing”, it’s the opposite. The companies that aren’t testing are losing money because they’re putting losers online without looking at the numbers. Companies that believe they’re wasting money. @MARTIJNSCH #LAUNCHSCALE
- 12. Expecting Big Wins MISTAKE #3 They’re expecting to win the lottery by setting up their testing program. Truth is, most of the tests that you’ll be running will be a loser. But these are the biggest wins that you’ll get as they’ll teach you something. MISTAKE #4 Looking at your competitors might be useful to come up with new ideas for your backlog. But 99% of the time your competitors also won’t have a clue on what they’re doing. My Competitor is Doing X, so that’s why we’re testing X. @MARTIJNSCH #LAUNCHSCALE
- 13. Page Running tests when you don’t have traffic. MISTAKE #5 When you don't have enough traffic you won’t reach the right significance (%95+) to make sure that you can say that you have a winner/loser or not. 13 @MARTIJNSCH #LAUNCHSCALE
- 14. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Hypothesis Do you really know what you will be testing? Write a hypothesis, you want to know why you’re testing what you’re testing. Otherwise you cannot actively make sure that you’re test are having the outcome that you’re expecting. It will also help you in making sure that your tests will provide an active learning instead of just trying to hunt down some results. PROCESSES ARE ESSENTIAL Ideas Quality Assurance Design & Engineering HypothesisRepeat A/B Testing Repeat this process… Run & Analysis 14 @MARTIJNSCH #LAUNCHSCALE
- 15. Page They don’t create a hypothesis. MISTAKE #6 When you start testing you want to make sure that you know what kind of impact you’re expecting from an experiment. 15 @MARTIJNSCH #LAUNCHSCALE
- 16. Testing multiple variables at the same time, making 3 changes basically requires 3 tests. MISTAKE #7 “Let’s test the button, oh and let’s also change the header + CTA”. MISTAKE #8 “Let’s run a test because I like the colors green and yellow more and I heard on BART that they work better”. Use numbers as the basis of your research, not your gut feeling. @MARTIJNSCH #LAUNCHSCALE
- 17. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Design & Engineering Have designers & engineers work together to create treatments When you’re designing and engineering your tests, because in all honesty who’s using the WYSIWYG editors after a few tests you want to make sure you’re in constant sync with your designers to make sure that the tests that you’re running are within the guidelines of your design (or not) and that the code is written as it’s supposed to be. PROCESSES ARE ESSENTIAL Ideas Quality Assurance Design & Engineering HypothesisRepeat A/B Testing Repeat this process… Run & Analysis 17 @MARTIJNSCH #LAUNCHSCALE
- 18. Before and After is not an A/B test. We launched, let’s see what the impact is. MISTAKE #9 A/B Testing doesn’t work when you’re just launching a feature and trying to compare the results: before/after. It doesn’t take into account all the different variables that could be impacting your results. MISTAKE #10 Many times I’ve seen companies or designers worry too much about the designs that they’re testing. Make sure that you can test with multiple variants if that’s possible so it’s not up to the designer to decide what is working. They go over 71616 revisions for the design. @MARTIJNSCH #LAUNCHSCALE
- 19. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Quality Assurance Are you sure your test is working? You have the test ‘working’ as your designers and engineers have created the deliverables for the test. But are you really sure that your test is working in all the browsers that your visitors are using? Make sure to test extensively before every test that you’re activating. PROCESSES ARE ESSENTIAL Quality Assurance Design & Engineering HypothesisRepeat A/B Testing Repeat this process… Ideas Run & Analysis 19 @MARTIJNSCH #LAUNCHSCALE
- 20. Page They don’t Q&A their tests. Even your mother can have an opinion this time. MISTAKE #11 Make sure that you’re QA’ing your experiment before you activate it. It will prevent you from stopping a test halfway when your results are skewed. #lovemoms 20 @MARTIJNSCH #LAUNCHSCALE
- 21. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Run & Analysis Time to have users experience your treatments It’s time to start running your test, make sure that you’re running. Essential is that you’re running your tests for the right duration. After this comest the best part, analysis. PROCESSES ARE ESSENTIAL Quality Assurance Design & Engineering Hypothesis Run & Analysis Repeat A/B Testing Repeat this process… Ideas 21 @MARTIJNSCH #LAUNCHSCALE
- 22. Page Running your tests not long enough, calling the results early. MISTAKE #12 You want to make sure that you reach significance, but at the same time you also need to make sure that you’re dealing with your business cycles. 22 @MARTIJNSCH #LAUNCHSCALE
- 23. Page Running multiple tests with overlap.. it’s possible, but segment the sh*t out of your tests. MISTAKE #13 You have multiple tests running? Great, but make sure that you also segment based on that data and plot what the impact is of the overlap in between tests. 23 @MARTIJNSCH #LAUNCHSCALE
- 25. Data is not sent to your main analytics tool, or you’re comparing your a/b testing tool to analytics, good luck. MISTAKE #14 MISTAKE #15 Basic fail #1, as long as you don’t have the right significance you’re basically running your testing program like flipping a coin: you have a 50/50 chance of finding a winner. Going with your results without significance. @MARTIJNSCH #LAUNCHSCALE
- 26. Page WHAT CAN YOU TEST? You want to know what might improve the bottomline. Repeat Start all over again: eat, test, sleep, repeat You’ve been running your first experiments that hopefully have been successful. Now comes the next step: repeating the process. PROCESSES ARE ESSENTIAL Quality Assurance Design & Engineering Hypothesis Run & Analysis Repeat A/B Testing Repeat this process… Ideas 26 @MARTIJNSCH #LAUNCHSCALE
- 27. Not deploying your winner fast enough, takes 2 months to launch MISTAKE #16 MISTAKE #17 Keep track of what you’re testing. Once you’re exceeding 50 experiments it’s unlikely that you still know what you’ve tested before and what the results were. They’re not keeping track of their tests. No documentation. @MARTIJNSCH #LAUNCHSCALE Speed is one of the most essential components of a Growth process. So not launching your winning tests or waiting for it for a while is not helping you or your company.
- 28. Page They give up. MISTAKE #18 NEVER EVER GIVE UP ON TESTING. 28 @MARTIJNSCH #LAUNCHSCALE
- 29. If you double the number of experiments you’re going to double your inventiveness - Jeff Bezos, Amazon
- 30. Page Ideas: 1. They just launch, they don’t test. 2. Companies that believe they’re wasting money. 3. Expecting Big Wins. 4. My Competitor is Doing X, so that’s why we’re testing X. 5. Running tests when you don’t have traffic. Hypothesis: 1. They don’t create a hypothesis. 2. Testing multiple variables at the same time, making 3 changes basically requires 3 tests. 3. Use numbers as the basis of your research, not your gut feeling. Design & Engineering: 1. Before and After is not an A/B test. We launched, let’s see what the impact is. 2. They go over 71616 revisions for the design. 30 Summary @MARTIJNSCH #LAUNCHSCALE
- 31. Page Quality Assurance: 1. They don’t Q&A their tests. Even your mother can have an opinion this time. Run & Analysis: 1. Running your tests not long enough, calling the results early. 2. Running multiple tests with overlap.. it’s possible, but segment the sh*t out of your tests. 3. Data is not sent to your main analytics tool, or you’re comparing your a/b testing tool to analytics, good luck. 4. Going with your results without significance 5. You run your tests for too long… more than 4 weeks is not to be advise, cookie deletion. Multiple issues. Repeat: 1. Not deploying your winner fast enough, takes 2 months to launch 2. They’re not keeping track of their tests. No documentation. 3. They Give Up! 31 Summary @MARTIJNSCH #LAUNCHSCALE
- 32. Page Thank you, have fun! 32 @MartijnSch #LAUNCHSCALE @MARTIJNSCH