Columbus Data & Analytics Wednesdays - June 2024
0 likes227 views
The document discusses the principles and practices of A/B testing, emphasizing the importance of hypotheses and statistical significance in experiment design. It contrasts frequentist and Bayesian approaches to data analysis, highlighting the advantages of Bayesian methods in terms of reliability and ease of implementation. Additionally, it addresses the complexities of testing in the context of privacy laws and the use of paid participant pools for data collection.
Columbus Data & Analytics Wednesdays - June 2024
- 1. Running experiments - from design to analysis A/B tests under the hood Maria Copot - CbusDAW
- 2. Who am I? Researcher at Ohio State (Dept of Linguistics) Main interest: characterising speakers’ knowledge of the words of their language Broader interest in quantitative methodologies and statistical analysis
- 3. …Why am I here? We both collect data and run experiments!
- 4. The pitch Digital analysts run experiments, software drives design and implementation Privacy laws, Cookie Ban™ - consequences for experimental data and design A peek under the hood of common software, and proposals for branching out
- 5. The next half hour of your life 1. The gears of A/B testing software 2. The bene fi ts of Bayesian methods 3. Paid participants - Proli fi c.co, Mechanical Turk
- 6. The goal vs the tools All statistical analysis and experimentation is in the service of a goal. Never lose track of the bigger picture! Is A/B testing what you need?
- 7. When to use A/B testing Comparing pairs of options differing along one dimension Compare multiple dimensions between pairs of options? Multivariate tests! Compare holistically di ff erent page designs? Redirect tests!
- 8. A/B testing Goal Hypothesis Choose variations Results! Mysterious A/B testing black box (…goblins?)
- 9. The importance of having a hypothesis Testing for random features might sometimes increase clicks or engagement but…
- 10. The importance of having a hypothesis Important to start from what you think is missing and why it will improve things. “Users prefer X to Y because Z” - Identifying areas for improvement helps come up with tests (X vs Y) - Both positive and negative outcomes teach us about user motivations (is Z true?)
- 11. The importance of having a hypothesis
- 12. Results do not speak on their own The narrative behind them is crucial for decisions
- 13. Statistical signi fi cance A/B test - is CR higher with a blue or a red button? Users CR P-value Blue (old) 1000 15% 0.006 Red (new) 1000 18%
- 14. 15% 18% p = 0.006 Statistical signi fi cance Assume no di ff erence between the blue and red button How likely is the red button observation? P value = probability of observation assuming no difference
- 15. Statistical signi fi cance Threshold for signi fi cance usually set at p < 0.05 (arbitrary value)
- 16. Statistical signi fi cance Threshold for signi fi cance usually set at p < 0.05 (arbitrary value) Can lower threshold for more precision, depending on application and variance in user behaviour. ✨Nothing magical about 0.05 ✨
- 17. Statistical signi fi cance Always look at both effect size and p-value! Larger expected effect sizes, easier to get low p-values Effect size: the difference between the means of the two hypotheses
- 18. The dangers of working with means
- 19. P-values and sample size Low p-values are facilitated by large sample sizes Users Conversion rate P-value Blue (old) 10 70% 0.69 Red (new) 10 80% Users Conversion rate P-value Blue (old) 100 70% 0.05 Red (new) 100 80%
- 20. P-values and sample size Users Conversion rate P-value Blue (old) 100000000 80.000 0.03 Red (new) 100000000 80.001 Power analysis how many data points are needed to have an 80% chance to discover an effect of the anticipated size?
- 21. P-values, sample sizes and time Running assumption: users split 50/50 between variations. But for certain features, not possible. Example: purchase totals for people using apple pay vs manually entering card details.
- 22. P-value Effect size (bigger difference = less overlap between hypotheses) Sample size (higher = more certainty in the difference)
- 23. A/B testing - sneaking in frequentist statistics Frequentists ask: how likely is the observation if the null hypothesis is true? Probability of the data given the (null) hypothesis
- 24. Bayesian A/B testing Bayesians ask: how likely is my hypothesis, given the data?
- 25. Bayesians vs frequentists Frequentist estimation a single unknown number with uncertainty around it, tested against the null hypothesis Bayesian estimation the entire distribution of the parameters of interest
- 26. Bayesian updating posterior likelihood prior marginal p(hypothesis|evidence) = p(evidence|hypothesis) * p(hypothesis) p(evidence) Out of all the times people press the button, how many were blue vs red?
- 27. Bayesian updating p(hypothesis|evidence) = p(evidence|hypothesis) * p(hypothesis) p(evidence) posterior likelihood prior marginal
- 28. Advantages of Bayesian methods • Intuitive results (p-values and con fi dence intervals are misunderstood) • Reliable even for small sample sizes (no need to pre-de fi ne sample sizes) • No need to estimate the e ff ect size in advance • Early stopping is allowed (continuous updating) • Faster pipeline to decisions • Can incorporate domain knowledge through priors • Estimates entire distribution
- 29. Testing features in the age of privacy laws Desired tests are often more complex and nuanced than “red vs blue button” • Longitudinal tracking • Multiple outcomes Need to know what behaviour comes from the same user
- 30. Paid participant pools Platforms like Proli fi c.co, MTurk, Qualtrics Panel (and others!) Consent forms allow you to track participant behaviour in depth Participants can be recontacted for follow-up qualitative assessments Large participant pool fi ltrable by detailed demographic information Cons: participants must be paid and know they are taking part in an experiment
- 31. Thank you! Any further questions: maria.copot.s@gmail.com