Taming Uncertainty: Planning Robust A/B-Testing
1 like456 views
The document discusses the importance of A/B testing and highlights common pitfalls, such as Simpson's paradox and false positives/negatives, when evaluating conversion results. It emphasizes the value of conducting randomized, double-blind controlled trials to minimize biases and improve the reliability of testing outcomes. Additionally, it offers practical tips for designing experiments and interpreting results to achieve more accurate conclusions.
Taming Uncertainty: Planning Robust A/B-Testing
- 1. W www.axelisys.co.uk T @axelisys Taming Uncertainty Robust, Practical A/B-Testing
- 2. W www.axelisys.co.uk T @axelisys Pitfalls of A/B-Tests 1 of 7 A/B-Test Uplift Conversion Rate Specialists CRO Agencies 33% Moving to A/B/n-Tests Too Early CRO is only part of it. (Source: VWO) MONEY… MONEY!MONEY…Ultimately all about the
- 3. W www.axelisys.co.uk T @axelisys Some Very Common Pitfalls Simpson’s Paradox Significance Testing Fail to Reject Null Vs Accepting Alternate False Positives & Negatives ‘‘I think it, therefore it’s true’’ (Cognitive Bias)
- 4. W www.axelisys.co.uk T @axelisys Simpsons Paradox Definition Two or more results concluded, but the combination contradicts that result (or vice versa). Compounded by over-fitting! “More people access the web on mobile devices than any other platform… …Except that older people don’t!”
- 5. W www.axelisys.co.uk T @axelisys False Positives & Negatives Type 1 and Type 2 Error Type 1 – Rejecting the Null When it is TRUE Chasing Rainbows Type 2 – Accepting the Null when it is FALSE Missing Rainbows & the Pots of Gold at the end!
- 6. W www.axelisys.co.uk T @axelisys QUIZ: RARE DISEASE “You are worried you might have icandozis and get tested. Suppose the testing methods for icandozis are correct 99% of the time. Icandozis is actually quite rare, occurring randomly in the general population in one in 10,000 people. If you test positive, what are your chances that you actually have icandozis?” a) 99% b) 90% c) 10% d) 1%
- 7. W www.axelisys.co.uk T @axelisys RANDOMISED, DOUBLE – BLIND CONTROLLED TRIAL Gold Standard:
- 8. W www.axelisys.co.uk T @axelisys Definition: Randomised Two+ Independent Groupings Randomly Assigned Subjects in your target group. “Randomly assigning subjects to all your categories is most likely to provide a representative sample”
- 9. W www.axelisys.co.uk T @axelisys Definition: Double Blind Researcher & Subject Don’t know which group Subject are assigned to. So researcher and subject behave the same for A and B tests. TIP: Automated allocation (e.g. Google Analytics) Image via ’John the Math Guy’ “Double Blind Studies Prevent either the Researcher or Subject Biasing the Results”
- 10. W www.axelisys.co.uk T @axelisys Definition: Controlled Every potential factor is fixed aside from the factor under test. Controlling for Covariates Minimises ‘confounding variables’ It’s currently raining AND they stand near puddles It’s not currently raining AND they stand near puddles Covariates are the puddles! Image via ‘Not the average’ blog “If someone goes outside and gets wet, does it mean it’s raining?”
- 11. W www.axelisys.co.uk T @axelisys DESIGNING EXPERIMENTS Top-5 Tips
- 12. W www.axelisys.co.uk T @axelisys 1. Develop A Hypothesis Start with a Testable Hypothesis Know the Question you want to Answer Make sure you can prove it’s false! (Karl Popper) Falsifiability = “Nullifiability ” Provides a NULL Hypothesis Analyse Results, Conclude AND …ACT! “"it is not only not right, it is not even wrong!"” - Wolfgang Pauli Physicist
- 13. W www.axelisys.co.uk T @axelisys 2. Develop a Solid Comparison Control Group = Baseline Performer (B-test) Experimental Group is A- test Randomly Allocate Subjects to Control & Experimental Group “In a poll, 97% of 783 GPs admitted that they had recommended a sugar pill or a treatment with no established efficacy for the ailment their patient came in with.” - PLOS Study 2013
- 14. W www.axelisys.co.uk T @axelisys 3. Account for Covariates …as much as possible Anything you can’t eradicate, limits results Option for further hypotheses/experiments Meaning dry puddles BEFORE! :) …Or use ANCOVA (Analysis of Covariance) After
- 15. W www.axelisys.co.uk T @axelisys 3 2 Quiz: Multiple Choice 1 Which is the Best Next A/B Pair? Null: “Red CTA No Better Than Green CTA”
- 16. W www.axelisys.co.uk T @axelisys 4. Compare with Significance Level How Confident Are you? Chi-squared most widely used 95% confidence intervals Degrees of freedom = number of categories -1 One tailed – Significance exclusively more or less than chi-square OR Two tailed – Expect either more or less than chi-square
- 17. W www.axelisys.co.uk T @axelisys 5. Comprehensively Experiment ALWAYS NOTE Include All Days of Week Days of Month etc. Making Decisions early risks losing full picture of engagement!
- 18. W www.axelisys.co.uk T @axelisys Lightening Experience: Twitter Engagement Tweet text (note, several users 1 tweet) Initial Expected Proportions Actual Proportions Favourites Retweets Engagement Hypothesis A test (find small-P) B test Total (n=) A B A B A B A B A B Null: Direct selling doesn't increase engagement. Sales message "Delighted to welcome {users}! Hope you have a great 2015! We're at www.ducredit.co.uk if U need us!" Friendly message "Thanks for following {users}! Here's hoping you start 2015 on a high! :)" 100 0.5 0.5 50 50 2 12 0 0 2 12 Null: Thankful message doesn't significantly alter engagement v welcome Huge #WELCOME to {users}! 2015 started well for you so far? We're thankful {users} joined us on the journey! Hope you have a great day! :) #welcome 88 0.5 0.5 63 25 21 12 6 5 27 17 Null: Shout out without 'joining us' has no effect Huge #WELCOME to {users}! Thank you for joining us on the journey! Huge shout-out to {users}! Hope you have a great day! Thanks :) #welcome 56 0.5 0.5 31 25 7 9 2 4 9 13 Null: Directing to FB page doesn't reduce favourites Huge shout-out to {users}! Hope you have a great day! Thanks :) #welcome Thanks to new followers {users}! Have a FB page? Like us on www.facebook.com/DuCredit #welcome 82 0.5 0.5 48 34 6 4 5 0 11 4 Null: New, funky welcome tweets don't increase engagement A MAHOOSIVE Welcome to {users}! Thanks for following! :) #welcome Huge shout-out to {users}! Hope you have a great day! Thanks :) #welcome 48 0.5 0.5 33 15 12 8 3 2 15 10 Validation Chart Unfollowers.com automated welcome tweets. A/B-testing engagement
- 19. W www.axelisys.co.uk T @axelisys Follow This Guy!... Ton Wesseling Ton Wesseling @tonw Chief Optimization Officer Testing Agency http://testing.agency/ http://www.slideshare.net/tonwesseling Doesn’t talk much, but when he does, it’s always solid!
- 20. W www.axelisys.co.uk T @axelisys Thanks for Viewing Further Reading 12 Common A/B-Testing Errors http://conversionxl.com/12-ab-split-testing-mistakes-i-see-businesses-make-all-the-time/ Random Variables and Probability Distributions https://www.khanacademy.org/math/probability/random-variables- topic/random_variables_prob_dist/v/random-variables (Khan Academy) Confidence Intervals http://en.wikipedia.org/wiki/Confidence_interval Normal Distribution http://en.wikipedia.org/wiki/Normal_distribution “Correlation & Dependence” Wikipedia http://en.wikipedia.org/wiki/Correlation_and_dependence Factor Analysis http://en.wikipedia.org/wiki/Factor_analysis Genome Biology http://genomebiology.com/ (Publishes research, software and new methods) Ethar Alali @EtharUK @Axelisys Managing Director & Chief Architect Polymath-MathMo. Programming since 9 years old. TOGAF 9 Certified, change agent. Blog: GoadingtheITGeek.blogspot.co.uk Specialist ICT Strategists & Advisors for some of the biggest household and corporate multi-nationals. Accredited Growth Voucher Advisors 2014/15 certified to deliver IT & Web Growth Consultancy as part of the government’s Growth Voucher Scheme. About Us Accreditations & Associations
Editor's Notes
- #4: There are many type of research and testing error that can be made. Each of which has the potential to cause your startup to miss opportunities or waste money, even if you run A/B-tests. After all, they aren’t magically not exposed to the same problems plaguing other research methods.
- #11: Covariates are variables which are likely to have a similar effect on the dependent variables (often your result – conversion rate, profit margin etc.) to the main variables under tests. Declaring a result without accounting for covariates leaves you open to confounding factors ruining your result and costing you money when acting on this wrong thing!
- #14: Anyone know what this effect on the right is called? This is one of the main reasons we campare against a control. Too many people individually see results which are not in any way proven. Homeopathy, astrology etc. are topics which have been highly controversial in the past, but the material itself has no scientific efficacy, even if there is efficacy because people ‘recover’ through the placebo effect.
- #17: Your result tells you what it is. As a rule of thumb, crucially, confidence intervals tell you how SURE you are!