A/B testing from basic concepts to advanced techniques
- 1. A/B Testing: from key concepts to advanced techniques By Anatoly Vuets Head of Analytics Letyshops
- 2. Agenda A/B test as part of statistical learning Basic ideas: H0, significance, power, etc. Common mistakes and misunderstandings Typical problems and solutions Bayesian approach to A/B testing Questions and discussion
- 3. A/B testing as part of statistical learning Prediction Inference Question What would be our best guess for a new data point yn+1 ? Question about the population Y: y1..n â Y stated in the form of binary hypothesis Answer Estimated value for yn+1 Accepting or rejecting the Null Hypothesis H0 Known answers Observable Unobservable Way to find the answer Model which is trained to minimize chosen loss function Binary decision based on comparing test statistics value and a threshold. The threshold is chosen to meet desired error tolerance
- 4. A/B testing as part of statistical learning Inference
- 5. Basic ideas: H0/H1, power, significance level etc.
- 6. Basic ideas: H0/H1, power, significance level etc.
- 7. Basic ideas: H0/H1, power, significance level etc. H0 H1 H0 1 - sign.level Error T1 sign.level H1 Error T2 1 - power power Test T(s) Truth
- 10. Common mistakes and misunderstandings 1 Starting A-B test without design 2 Early stopping 3 P-value interpretation P.value is not a âprobability that B is better then Aâ
- 11. How to interpret results in a âsimpler wayâ Prior beliefs: Likelihoods: Posteriors:
- 12. How to combine results from different experiments
- 13. What is the probability that the repeated experiment wont reproduce results? About 72%! Can be increased to about 90%
- 14. Sample size reduction Stratification CUPED
- 15. Bayesian testing 1. Set you prior beliefs about test statistics as a prior distribution 2. Collect some data 3. Get posterior distribution of the test statistics using Bayesâ theorem 4. Compute expected error 5. Repeat steps 3-5 till expected error drops below a threshold Bayesian approach to the inference problem
- 16. Different approaches: Bayesian testing
- 17. Different approaches: Bayesian testing
- 18. Summary and questions Frequentist framework for A/B testing is more mature overall as it provides control of errors and well-defined stopping rule. Bayesian framework helps a lot to alleviate practical issues which occur when using frequentist A/B testing