Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data
0 likes719 views
This document discusses best practices for running A/B tests, including collecting enough data over multiple weeks and business cycles, avoiding relying solely on significance levels to determine a winner, and integrating test data with web analytics. It also emphasizes formulating a clear hypothesis by outlining why a change is needed and how its impact will be measured. The presenter advocates getting familiar with key statistical concepts and learning statistics.
![1. Because we saw [data/feedback]
2. We expect that [change] will cause [impact]
3. We’ll measure this using [data metric]](https://image.slidesharecdn.com/michaelaagaardpracticalabtestingstatistics-151210212510/85/Practical-A-B-Testing-Statistics-5-Tips-To-Help-You-Get-Reliable-Data-32-320.jpg)
![Because [exit surveys indicated that saving money is
most important to our users.]
We expect that [tweaking our value prop to reflect this]
will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our
key metric.]](https://image.slidesharecdn.com/michaelaagaardpracticalabtestingstatistics-151210212510/85/Practical-A-B-Testing-Statistics-5-Tips-To-Help-You-Get-Reliable-Data-33-320.jpg)
![Because [exit surveys indicated that saving money is
most important to our users.]
We expect that [tweaking our value prop to reflect this]
will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our
key metric.]](https://image.slidesharecdn.com/michaelaagaardpracticalabtestingstatistics-151210212510/85/Practical-A-B-Testing-Statistics-5-Tips-To-Help-You-Get-Reliable-Data-34-320.jpg)
![Because [exit surveys indicated that saving money is
most important to our users.]
We expect that [tweaking our value prop to reflect this]
will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our
key metric.]](https://image.slidesharecdn.com/michaelaagaardpracticalabtestingstatistics-151210212510/85/Practical-A-B-Testing-Statistics-5-Tips-To-Help-You-Get-Reliable-Data-35-320.jpg)
![Because [exit surveys indicated that saving money is
most important to our users.]
We expect that [tweaking our value prop to reflect this]
will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our
key metric.]
We expect to see reliable results in [four weeks.]](https://image.slidesharecdn.com/michaelaagaardpracticalabtestingstatistics-151210212510/85/Practical-A-B-Testing-Statistics-5-Tips-To-Help-You-Get-Reliable-Data-36-320.jpg)
Practical A/B Testing Statistics: 5 Tips To Help You Get Reliable Data
- 2. Your%Presenter Michael%Aagaard 4 Senior%Conversion%Op:mizer%at%Unbounce% 4 Full4:me%CRO%for%the%lat%seven%years% 4 Interna:onal%speaker% 4 Founder%of%ContentVerve.com
- 4. “When can I stop my A/B Test?”
- 5. “Why run an A/B Test?”
- 6. To get a representative picture of how a variant would perform if implemented permanently “in the wild”.
- 7. To get a representative picture of how a variant would perform if implemented permanently “in the wild”.
- 8. Do Not Use 95% Significance Level as a Stopping Rule 1
- 9. After 1 day +151.4% / significance 97% 655 visitors / 18 conversions
- 10. After 2 days +98.4% / significance 95% 988 visitors / 23 conversions
- 11. After 1 week +0.2% / significance 50% 3031 visitors / 58 conversions
- 12. After 2 weeks +18.1% / significance 77% 5277 visitors / 78 conversions
- 13. After 4 weeks +36.2% / significance 96% Total sample: 9396 visitors / 122 conversions
- 14. What we have now: 9,396 What we need: 34,304 visitors in total What we’re missing: 24,908 visitors 73% of the required sample size. http://www.evanmiller.org/ab-testing/sample-size.html
- 16. “Formulas don’t know whether they are being used properly, and they don’t warn you when your results are incorrect.” - Deborah J. Rumsey, PhD
- 17. “Real accuracy depends on the quality of data as well as on the sample size.” - Deborah J. Rumsey, PhD
- 18. Worth Getting Familiar With: - Power Level - P-Value - Margin of Error - Confidence Interval - Null hypothesis / alternative hypothesis
- 19. Make Sure You Collect Enough Data to Yield Meaningful Insight 2
- 20. Full weeks Full business cycles
- 22. Full weeks Full business cycles Week One Week Two Week Three Week Four
- 23. Test duration: 2-4 weeks (2 or more business cycles) Sample size: Pre-calculated sample size (users) 400+ conversions (goals) Significance: 95% or higher Power: 80% P-Value: 0% What I look for:
- 24. Be Careful With Averages - They Hide the Truth 3
- 25. Devices Browsers Channels Country Days of the week Hours of the day User types Motivation Campaigns Age Gender Gender BrowsersDevices Browsers Channels Countries Days of the week Hours of the day User types Motivation Campaigns Age Days of the month Gender Devices Experiment
- 28. Integrate Test Data with Your Web Analytics Setup And Get the Full Picture
- 30. Formulate a Clear Test Hypothesis 4
- 31. Why do we think we need to make a change? What is it that we want to change? What impact do we expect to see? How will we measure this impact?
- 32. 1. Because we saw [data/feedback] 2. We expect that [change] will cause [impact] 3. We’ll measure this using [data metric]
- 33. Because [exit surveys indicated that saving money is most important to our users.] We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.] We’ll measure this using [form submission rate as our key metric.]
- 34. Because [exit surveys indicated that saving money is most important to our users.] We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.] We’ll measure this using [form submission rate as our key metric.]
- 35. Because [exit surveys indicated that saving money is most important to our users.] We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.] We’ll measure this using [form submission rate as our key metric.]
- 36. Because [exit surveys indicated that saving money is most important to our users.] We expect that [tweaking our value prop to reflect this] will cause [more people to fill out our lead form.] We’ll measure this using [form submission rate as our key metric.] We expect to see reliable results in [four weeks.]
- 39. Live%Q&A%Time