A/B testing powered by Big data - Saurabh Goyal, Booking.com
0 likes335 views
The document discusses the implementation of A/B testing at Booking.com using big data to enhance user decision-making. It outlines key components including hypothesis building, metrics for success, and tools utilized to analyze data. The author emphasizes the importance of learning from both successful and unsuccessful experiments to improve business outcomes.
A/B testing powered by Big data - Saurabh Goyal, Booking.com
- 1. A/B Testing powered by Big Data in Booking.com Saurabh Goyal | February 2018 1
- 2. Structure ● A/B testing ● Big Data ● Example
- 3. A/B testing What is it? 3
- 4. Which button color is the best? OR ? 4
- 6. Experiment flow. 50% 50% 10% conversion 20% conversion Win! Lose. 6
- 7. 10% conversion 20% conversion What does “the best” mean? 50% 50% Win! Lose. 7
- 8. Metrics: ● Conversion ● Click-through rate ● Logins ● Bounce rate ... anything your business wants the customers to do! 8
- 9. Additional metrics: ● Clicked on an element ● Clicked on a different element ● Spent time on the page ● Scrolled down Helps to learn about user behavior. 9
- 11. “If I implement X, the users will do Y, and it’s good because Z” 11
- 12. “If I... add a cat to my presentation, the listeners will... say AAAaaw, and it’s good because... they will remember my talk” 12
- 13. 13
- 14. Success: Congratulate yourself! Continue on topic. Where else can you apply the same approach? 14
- 15. Failure: That’s unfortunate. But don’t give up just yet! Analyze what went wrong. 15
- 16. Repeat. 16
- 17. 17
- 18. RepeatAnalysisExperimentMetrics Steps to a successful experiment: Hypothesis 18
- 19. Learning is the most important thing 19
- 20. A/B testing: not only button colors 20
- 21. Big Data
- 23. ● Over a million properties ● 1000 Inventory events per minute. ● 3.9 TB data from Inventory events. 23 Data Volume
- 24. ● Mysql replicator ● Kafka ● Hbase ● RTM ● Oozie 24 Tech used
- 26. Example 26
- 27. User research 27
- 28. Brainstorm 28
- 32. Metrics 32
- 34. Thank you! Saurabh Goyal | February 2017 34
- 35. Questions? 35