DN18 | A/B Testing: Lessons Learned | Dan McKinley | Mailchimp
0 likes200 views
The document discusses the complexities of causal inference and experimentation in data-driven contexts, highlighting challenges like correctness, effectiveness, and the introduction of vendor motives in experimentation. It emphasizes the importance of running multiple experiments to discern effective strategies and addresses the subtleties involved in coding and adopting new methodologies. The author suggests that cross-disciplinary expertise is crucial for executing product experiments, ultimately advocating for a systematic approach to experimentation within organizations.
DN18 | A/B Testing: Lessons Learned | Dan McKinley | Mailchimp
- 1. CAUSAL INFERENCE for FUN and PROFIT @mcfunley Data Natives Berlin 2018
- 2. CAUSAL INFERENCE for FUN and PROFIT @mcfunley Data Natives Berlin 2018
- 5. xkcd.com/208 please give him all your money Trying science:
- 7. xkcd.com/208 please give him all your money Trying science: Kind of complicated, tbh
- 8. xkcd.com/208 please give him all your money Trying science: Kind of complicated, tbh I have now done this twice
- 10. Experiments?
- 11. user
- 12. A B C user
- 15. A B C
- 18. Sources of chaos: 1. It is tricky to do any of this correctly, turns out
- 19. Sources of chaos: 1. It is tricky to do any of this correctly, turns out 2. You are introducing the possibility that you will unlaunch things
- 20. The idea had been spoken. And the words wouldn’t go back after they’d been uttered aloud.
- 21. Things to worry about
- 22. Correctness
- 23. Effectiveness
- 24. Adoption
- 25. Use a vendor
- 27. Confidence, if constrained and directed tastefully, is not a bad thing.
- 29. Things that are hard: (non-exhaustive)
- 30. Counting things
- 31. Counting things (that arrive quickly or in large numbers or both)
- 32. Bucketing
- 33. The Stats Model
- 34. The Stats Model
- 35. IMPORTANT CAVEAT javascript testing is bad
- 36. IMPORTANT CAVEAT vendors are independent humans with motives
- 37. cool-testing-platform.io Advanced settings you probably don’t need to worry about lol 90% Significance level It’s cool trust me This is fineCancel
- 38. VENDOR MOTIVES: align with high experiment volume
- 39. Pavel Sokov
- 40. Ease into it
- 41. Gopal Vijayaraghavan One theory on experiment volume
- 42. Gopal Vijayaraghavan Individually small chances of finding a real effect
- 43. The “loads of experiments” theory: Knowing where to run experiments is a problem.
- 44. The “loads of experiments” theory: Knowing where to run experiments is a problem. Knowing where to run experiments is the biggest problem.
- 45. The “loads of experiments” theory: Knowing where to run experiments is a problem. Knowing where to run experiments is the biggest problem. Running a lot of experiments is a way to find such a place.
- 46. The “loads of experiments” theory: Knowing where to run experiments is a problem. Knowing where to run experiments is the biggest problem. Running a lot of experiments is a way to find such a place. Running a lot of experiments is a good way to find such a place.
- 47. if bucket(user) and eligible(user): return render(‘new.tpl’) return render(‘old.tpl’) derp The biggest problem is subtle mistakes
- 50. STATS: oh wow much worse
- 51. EXperIMENTpOWER ODDS OF ANARCHY ARE UNCORRELATED TO
- 52. The “loads of experiments” theory: Knowing where to run experiments is a problem. Knowing where to run experiments is the biggest problem. Running a lot of experiments is a way to find such a place. Running a lot of experiments is a good way to find such a place.
- 58. Gopal Vijayaraghavan Ok, once you know what you’re doing
- 59. Changing process
- 61. class CogWidget { // stuff } $w = new CogWidget();
- 62. class CogWidget { // stuff } class CogWidget2 { // stuff // new stuff } $w = (bucket($user) == ‘control’) ? new CogWidget() : new CogWidget2();
- 63. “ ~ The problem with code re-use is that it gets in the way of changing your mind later on. tef https://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to
- 64. class CogWidget { // stuff } class CogWidget2 { // stuff // new stuff } $w = new CogWidget();
- 66. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 S M T W R F S meetings
- 67. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 S M T W R F S
- 68. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 S M T W R F S meetings
- 69. if bucket(user): return render(‘new.tpl’) return render(‘old.tpl’)
- 70. if has_no_edge_cases(user): if bucket(user): return render(‘new.tpl’) return render(‘old.tpl’)
- 71. IT’S EASY just change basically everything about development and add an entirely new technical discipline
- 72. OK NOT REALLY maybe start by building a single team with expertise
- 75. Concluding
- 76. Midway upon the journey of our life I found myself within a forest dark For the straightforward pathway had been lost
- 77. Correctness
- 78. YES NO A cross-disciplinary base of expertise for executing the full lifecycle of product experiments that can be built upon moving forward. A declarative YAML framework feeding realtime metrics into a Kafka cluster for self-adjusting multi-arm bandit testing.
- 79. Effectiveness
- 80. Adoption
- 81. “ ~ Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better… Douglas Bowman http://stopdesign.com/archive/2009/03/20/goodbye-google.html
- 82. CREDIBILITY Comes from success as well as avoiding egregious wastes of time
- 84. Humility