Camilo Martinez, Software Development Team Lead at Booking.com - The lifecycle of a machine learning product
- 1. Camilo Martinez Visitor Profiling Team Lead The lifecycle of a machine learning product
- 2. Goal. Goal of this talk: Share an approach to developing a machine learning product
- 3. The AI Flywheel Users give us (behavioral) data that help us build better models to improve the user experience
- 4. Data collection Start collecting data early and collect it often
- 5. Not so well. Well. Not so well. Next Quarter. Customer journey trace Search Property page Book attempt
- 6. Search data ● Number of adults ● Checkin date ● Filters selected ● Total number of properties ● Properties in this page ● Many more...
- 7. Data collection via Events Events System Events repo Custom workflow Flat view
- 8. Data collection via Streams Hotel impressions
- 9. Feature engineering Selecting useful features that are available in production
- 10. Features already available Easy to use features that are already implemented
- 11. Feature gathering Context sensitive, might rely on other features (recursively)
- 12. Feature gathering Context sensitive, might rely on other features (recursively)
- 13. Feature Store Standard repository of features
- 14. From hypothesis to tasks Could Machine learning help your users?
- 15. The 7 powers of machine learning Taken from Machine Learning @ Booking.com by Lucas Bernardi, Dennis Bohle, Onno Zoeter 1. Predict a fact about the Future 2. Describe an Alternative Future 3. Guess a fact about the Present 4. Materialize subjectivity 5. Untangle complexity 6. Automate human tasks 7. Optimize a Target Variable
- 16. Not so well. Well. Not so well. Next Quarter. Can we predict where the users want to go? Hypothesis Power Task Similar users like to go to similar destinations. If we know what the user has search in the past, Can we suggest some destinations where they would like to go? 3. Guess a fact about the Present
- 17. Not so well. Well. Not so well. Next Quarter. Can we predict where the users want to go? Hypothesis Power Task Similar users like to go to similar destinations. If we know what the user has search in the past, Can we suggest some destinations where they would like to go? 3. Guess a fact about the Present
- 18. Not so well. Well. Not so well. Next Quarter. Can we summarize the information available in the reviews? Hypothesis Power Task Reading reviews is time consuming. Is there a way we can summarize the information that they have? 5. Untangle complexity
- 19. Models tackle user problems Can we deliver models that are scalable and reusable?
- 20. Recommender Systems Repository of (proven) solutions ● Machine learning models that have been produced ● Products that use those models ● Model description and training description Inhouse Tool
- 21. How do we know if it is better for our users? A/B Testing
- 22. Base. Variant. Which one performed better?
- 23. Base. Variant. Which one performed better?
- 24. Is better accuracy an end in itself? When to invest and when to abandon
- 25. Accuracy vs. Reward Loss vs. KPI Taken from Machine Learning @ Booking.com by Lucas Bernardi, Dennis Bohle, Onno Zoeter
- 26. Taken from Machine Learning @ Booking.com by Lucas Bernardi, Dennis Bohle, Onno Zoeter Accuracy vs. Reward Loss vs. KPI
- 27. When to abandon? ● Saturation effect ● Shifts in traffic ● KPI is no longer sensitive to improvements to the metric you can manipulate
- 28. When to abandon? ● Even if the destinations recommended are better, users clicking in these destinations may not lead to actual bookings
- 29. When to invest? ● Leap of faith ● High potential reward ● Well established industry solution e.g. Deep Learning
- 30. How would you label this image?
- 31. Commercial solution ● Crib ● Baby bed ● Furniture ● Railing
- 32. Internal Deep Learning solution ● Sea view ● Balcony/Terrace ● Patio ● Seating area
- 33. Reality is messier Taken from Machine Learning @ Booking.com by Lucas Bernardi, Dennis Bohle, Onno Zoeter
- 34. Taken from Machine Learning @ Booking.com by Lucas Bernardi, Dennis Bohle, Onno Zoeter Invest? Reality is messier