A/B testing problems
Download as PPTX, PDF0 likes898 views
The document discusses the issues associated with A/B testing, particularly the reliance on statistical assumptions that can lead to misleading results. It highlights the discrepancies between expected and actual outcomes in various tests, emphasizing the need for careful analysis and replication. Additionally, the document touches on the importance of hypothesis formation and the potential pitfalls of multivariate testing and multiple comparisons.
A/B testing problems
- 1. A/B testing and problems with statistics Web Analytics Wednesday Singapore Nikolay Novozhilov, Wego.com www.novozhilov.co
- 2. Is there a problem with A/B testing?
- 3. Imaginary uplifts 100 tests done, 10 successful, 10% uplift each… …expect 159% growth! Expectation Reality
- 4. Why? … and what to do about it
- 5. Lies, damned lies, and statistics All different! All based on assumptions!!! Tool Test used Optimizely Two-tailed sequential likelihood ratio test with false discovery rate controls Google Analytics Bayes estimate with uniform beta prior VWO Intersection of confidence intervals for binominal distribution Leanplum Confidence intervals at p=5%, unknown statistic Usereffect Chi-square statistics Commerce Sciences Welch's t-test
- 6. What is p-value and why it is 5%? All tests are based on assumptions! Assumption #1: You don’t look at the data upfront
- 7. What happens if you look? I played Monte Carlo in Excel And here is the result: • 5% p-value • 1000 “users” in each sample • CR of 2% • A wins over A 29% of the times!
- 8. What do you do about it? Don’t look! (just kidding) Google “O'Brien & Fleming interim analysis” (no, still kidding ) Keep calm, more stuff coming!
- 9. “My test on Buy button showed interesting results…” Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! -3% -23% +6% -9% -2% +22% -11% -14% -1% +9% -12% -1% 10000 users in each variant, base CR=1%
- 10. But in reality all colors were the same… Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! Buy Now! -3% -23% +6% -9% -2% +22% -11% -14% -1% +9% -12% -1% 10000 users in each variant, base CR=1%
- 11. The real problem! Multivariate testing Multiple comparisons
- 12. Be smart or be Google Sample size Significa nce Effect size Power
- 13. Start with a good hypothesis! But people are good in finding plausible explanations for data!
- 14. Replication Do your dirty business Register Replicate This might work!
- 15. Stop math, I’m a web designer!
- 16. Visual way of doing it
- 17. Has some stat meaning! ReplicationsVariance observation
Editor's Notes
- #3: Ask audience: Do you do A/B testing? What you think is a problem?