Learn Like a Human: Taking Machine Learning from Batch to Real-Time
6 likes1,327 views
The presentation discusses the transition from batch to real-time machine learning, emphasizing the common challenges faced by publishers and retailers in optimizing user experiences. It introduces contextual bandits as a solution to improve revenue by dynamically serving content tailored to individual users, moving beyond traditional A/B testing. The approach leverages real-time data to continuously adapt and learn from user interactions, ensuring efficient and effective personalization.
Learn Like a Human: Taking Machine Learning from Batch to Real-Time
- 1. Learn Like a Human: Taking Machine Learning from Batch to Real-Time By Elad Rosenheim, Senior Software Architect delivered in
- 2. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • The Industry’s Problem • Old School Solutions • Meet the Contextual Bandits On the Agenda:
- 3. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • Publishers, retailers, SaaS all share a common problem • They know their domain but not how to optimize for each user • Screen real-estate is limited yet everyone sees the same thing The Industry’s Problem
- 4. Learn Like a Human: Taking Machine Learning from Batch to Real-Time Commonly asked questions • What articles should be shown on the homepage of an online publisher? • What titles and images would attract the most clicks? • What should the order and layout of the articles be? • Should every user see the “Top Videos” section? • What is the best layout for the homepage?
- 5. Learn Like a Human: Taking Machine Learning from Batch to Real-Time Commonly asked questions • Which product sorting layout would yield the highest revenue? • Should the layout differ significantly between different user segments?
- 6. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • What types of products should be shown, and to whom? Commonly asked questions
- 7. Learn Like a Human: Taking Machine Learning from Batch to Real-Time In the Beginning: • First came the educated guess • Then came the A/B test • "Data Beats Opinion“ • Freedom to experiment (with nice tools) • Hopefully: Less fear of change, less politics • How does it work? • Split traffic between baseline and alternative variations • In theory: Sit and wait for significant results • In practice: Peek at the numbers ‘til “95% confidence” is reached
- 8. Learn Like a Human: Taking Machine Learning from Batch to Real-Time While you wait, you're bleeding clicks! Clicks = Money What about the really dynamic stuff? Campaigns, headlines, products on sale A/B Tests: Already Old School?
- 9. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • A Single-arm Bandit • Suppose I have multiple arms in front of me, each with its unknown mean reward… • How do I optimize income from multiple machines? • Caution or Haste? • Explore vs. Exploit • In our context: How do I optimize multiple variations? Enter the Multi-arm Bandits
- 10. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • (Very) Simple Solutions • ε-greedy, ε–decreasing • First 100% random explore, then ~90% exploit? • Magic numbers, built-in revenue loss • Bayesian-based approaches • Smoother curve from explore to exploit • “Winner” is now a less relevant term Bandits - A Classic Problem
- 11. Learn Like a Human: Taking Machine Learning from Batch to Real-Time We want to find the variation “best on average“ …but we’re not improving the conversion rate of any single variation Bandits work well when… 0.4%1.7%2.4%
- 12. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • Each of us is a beautiful and unique feature vector! • By showing the right variation to the right people, we can improve conversions per variation and beat the best variation • Machine learning challenge accepted! Enter Personalization
- 13. Learn Like a Human: Taking Machine Learning from Batch to Real-Time The Usual Suspects • Collaborative Filtering? • Very big, very sparse matrix • Cold start, batch process • Not suitable in this case • Classifiers? • Logistic Regression, Random Forest et al. • Periodically learn over all converters so far • More data = more time, bigger model • “Partial Feedback” problem: • Users aren’t of class “Variation A” or “Variation B”. • Rather, for any given user, we’re placing our bets on a variation and don’t know what the reward would be for any other alternative.
- 14. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • Like a bandit, we need to learn as we go (not in batch), but this time with “context” - the user’s data • Incremental Learning over the stream of impressions & rewards (“Partial Fit”) • We’re looking to… • Start learning from the first impression • Handle the explore-exploit curve • Run fast (enough) • Worst case: Converge on the best variation like a bandit What We Need
- 15. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • They “eat” the data stream • They demand fast access to user data • Historical or immediate • Their model is always ready for action • In the Papers • Linear Bayes, LinUCB • What we do: Per-Variation Logistic Regression • A variant supporting updates in “mini-batches” • Exploration-on-top • Worst case: “Garbage In Multi Arm Bandit Out” • Light on memory, compact output Meet the Contextual Bandits
- 16. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • Online should be fast & scaled • Offline: A test-bed for iteratively testing new ideas • New algorithms • Tweaked parameters • Feature transformations - last but not least! How We Do It: Online & Offline
- 17. Learn Like a Human: Taking Machine Learning from Batch to Real-Time Persist Model The Online Flow DY Web Servers: a. Get our script b. Log impressions, conversions Learn Workers User DB Load to Predict Server Queue Per Test A B C A B C A B C Predictions
- 18. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • Test, Improve, Iterate • Using real-world data • Using generated data • From easy to hard The Offline Evaluator
- 19. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • Learn in the center site, predict quickly in each geo. How? • Push models via local Redis slaves • Compressed SSH tunnel • User data - daily aggregation • Store into LMDB (simple, fast memory-mapped K/V DB) • Sync via S3 (LZ4 compressed), read from local SSD • Learn & Predict services • Python as ML lingua franca: NumPy, SciPy, scikit-learn Going Global
- 20. Learn Like a Human: Taking Machine Learning from Batch to Real-Time • Better data beats better algorithms • Reduce aggressively Keep It Simple, Smart! By Elad Rosenheim, Senior Software Architect Final Word: * Thanks to Idan Michaeli, Lead Data Scientist
Editor's Notes
- #4: איך ניסו עד היום לתקוף את הבעיות האלה?
- #5: בואו ניקח את הפיתרון שלנו שלב אחד קדימה
- #6: בואו ניקח את הפיתרון שלנו שלב אחד קדימה
- #7: בואו ניקח את הפיתרון שלנו שלב אחד קדימה
- #8: בקיצור, העולם לא כ"כ מופלא כפי שמוכרים לנו
- #9: בואו ניקח את הפיתרון שלנו שלב אחד קדימה
- #10: לבעיה הזו אין פיתרון אופטימלי, אבל יש בהחלט גישות שונות ברמות שונות של מורכבות
- #11: עכשיו, בנדיטים הם לא כ"כ רעים למעשה...
- #12: והמגבלה הזו מביאה אותנו לשלב הבא בחיפוש, והוא: פרסונליזציה
- #13: יפה, אז קיבלנו את האתגר. על איזה אלגוריתמים אנחנו חושבים?
- #14: אז בעצם, אנחנו מחפשים משהו אחר – משפחה חדשה של אלגוריתמים
- #15: אז בואו ונכיר את הגיבורים החדשים שלנו...
- #16: אלגוריתם כשלעצמו זה דבר נחמד, אבל איך בונים את כל המעטפת לפרודקשן?
- #17: בואו ונבין טוב יותר את הזרימה