How A/B Tests Lie to Us and How to Drive Genuine Improvement
0 likes666 views
The document discusses the limitations and misconceptions surrounding A/B testing, highlighting how statistical significance can be misleading and the high chances of false-positives in experiments with too many variations. It emphasizes the importance of testing theories rather than superficial changes, and encourages seeking significant effects. Additionally, it warns against relying solely on crowd-sourced opinions, as they may stifle creativity.
How A/B Tests Lie to Us and How to Drive Genuine Improvement
- 1. SIX PUZZLES How A/B tests lie to us, and how to drive genuine improvement Jason Cohen, founder & CTO http://wpengine.com
- 3. 654 out of 24,483 visits 85% chance-to-beat
- 4. What do stats really tell us?
- 6. 654 out of 24,483 visits 85% chance-to-beat
- 7. Finding the Cheaters 105 athletes. 100 are clean. 5 are juicing. Test is 95% accurate.
- 8. Finding the Cheaters IN REALITY Clean (100) Juicing (5) TEST RESULT (95% accuracy) Clean Juicing 5 095 5
- 9. Insight: Rare events are indistinguishable from random noise
- 10. 758 out of 24,483 visits 85% chance-to-beat
- 11. Easy significance testing with hamsters Trust in the Hamster. bit.ly/abhamster
- 12. Easy significance testing N = A+B = total # of conversions.
 D = (A–B)/2 = half the difference.
 Significant if D2 > N. bit.ly/abhamster N = 758 + 685 = 1443
 D = ( 758 — 685 ) / 2 = 37
 1369 > 1443?
 No! Not significant.
- 13. Final Result: As predicted Wasted another month
- 14. 30 40 50 60 70 12/2/09 12/16/09 1/5/10 1/13/10 2/2/10 4/9/10 4/27/10 9/24/10 9/14/11 11/8/11 OtherInbox Splash Page
- 16. Lesson #1 Beware the Rare
- 17. Lesson #2 Trust the Hamster
- 19. 41 Shades of Blue
- 20. 41 Shades of Blue Does this... ...get more clicks than this?
- 21. 2 Shades of Blue with 1 winner at a 95% confidence level, chance of a false-positive: 5%
- 22. 3 Shades of Blue with 1 winner at a 95% confidence level, chance of a false-positive: 9%
- 23. 10 Shades of Blue with 1 winner at a 95% confidence level, chance of a false-positive: 40%
- 24. 41 Shades of Blue with 1 winner at a 95% confidence level, chance of a false-positive: 88%
- 26. Oops, you’re doing this.
- 28. Lesson #3 Test theories, not headlines
- 30. Bible Code!
- 31. Bible Code!
- 33. War and Peace
- 35. GA reveals ideas, not facts
- 36. Lesson #4 Exploration yields new theories, not new truth.
- 38. Headwinds to improvement significance → huge N → N/A high failure rate → many tests → false-positives rare events → worse on both!
- 39. Summary: Subtly is the Enemy
- 40. Insight: You don’t want “Subtle” anyway!
- 41. Lesson #5 Seek Huge Effects
- 42. Can you A/B test your way to big results?
- 44. Wisdom of the Crowd “I’m as clueless as you, but in the equal and opposite direction.”
- 45. What’s the funniest joke?
- 46. And the winner is…
- 47. Jokes, Pizza, Design, Features, Voice, Values, Culture
- 48. Lesson #6 Crowds find objective truth but kill creativity and greatness
- 49. Think for yourself. Test theories. Seek big effects. Jason Cohen, founder & CTO http://wpengine.com
- 50. Register now for our next webinar! “Your Workflow, Your Way, with WP Engine” Register now: http://wpeng.in/ywyw
- 51. Q & A Jason Cohen, founder & CTO http://wpengine.com