Real-Time Machine Learning at Industrial scale (University of Oxford, 9th Oct 2012)
12 likes5,006 views
The document discusses real-time machine learning at an industrial scale, highlighting the trade-offs between accuracy and latency in various use cases, including ecommerce, financial services, and media. It emphasizes the importance of hybrid solutions that combine both batch and real-time processing to optimize outcomes. Additionally, it outlines the challenges of finding skilled data scientists and the significance of employing distributed algorithms in big data applications.
Real-Time Machine Learning at Industrial scale (University of Oxford, 9th Oct 2012)
- 1. Real-Time Machine Learning at Industrial scale ... the battle of accuracy vs latency tumra.com @tumra 9th October 2012 TUMRA LTD, Building 3, Chiswick Park, 566 Chiswick High Road, W4 5YA Michael Cutler @cotdp
- 2. $ whoami Michael Cutler (@cotdp) ● Previously at British Sky Broadcasting ○ Last 7 years in R&D ○ Created several patented systems & algorithms ○ Kicked off ‘Big Data’ initiative at Sky in 2008 ● Co-founder CTO @ TUMRA in March '12 ○ Real-time big data science platform ○ Alpha-testing with selected clients
- 3. Agenda ● Background ● Real-Time vs Batch processing ● Accuracy vs Latency ● Use Cases ○ eCommerce ○ Financial Services ○ Media ● Questions
- 4. Background Big Data is "in vogue", but what does it mean: ● Distributed processing ● Massively scalable ● Commodity Apache Hadoop is "Kernel" of Big Data OS: ● Distributed Filesystem (HDFS) ● Parallel Processing (Map/Reduce, YARN)
- 5. Background (cont'd) Solving problems with Big Data is hard: ● Tools are all low-level (Pig, Hive etc.) ● Skills are hard to find What is "Data Science": ● Understanding data & solving problems ● Applies the following skills: ○ Statistical Analysis ○ Machine Learning ○ Communicating Results
- 7. Batch - Hoppers, Bins, Buckets Credit: http://bit.ly/Q71u4W
- 8. Real-Time - Flows & Streams Credit: http://bit.ly/NOslqf
- 9. Real-Time vs Batch processing Similarities to the Industrial Revolution: ● From handicraft to Batch & Real-Time ● Complexity increases Need for "Real-Time": ● Wherever the variation can change faster than you can retrain models ● When you can't pre-compute everything ahead of time
- 11. Accuracy vs Latency Netflix Prize winning entry :- ● Ensemble of 100's of models ● Massively compute intensive solution ● Marginally better than much simpler models IBM won the KDD Cup 2009 (Orange) :- ● IBM Watson team won by sheer brute force ● Used a "one of everything" approach generating hundreds of models
- 12. Accuracy vs Latency (cont'd) Mathematical navel-gazing: ● Often the factor we're optimising for, isn't the thing we measure improvement in: ○ User ratings vs. customer longevity/value ○ Overfitting outliers vs. missing clear Fraud Given the choice between a "best guess" now, and a "marginally better" answer later, I'd take the "best guess" every time.
- 13. However, that doesn't mean...
- 14. Accuracy vs Latency (cont'd) It's a trade-off: ● Sometimes "best guess" is good enough, ● Other times we can wait for the accuracy, ● And of course, occasionally we want both! Key objective: ● Most appropriate solution for the use-case ● Hybrid solutions part batch, part real-time
- 16. Use Case - eCommerce Objective - Increase profits How: ● Match potential customers to the right products ● Personalise user experience on web & email ● Customer lifecycle management Method: ● Ensemble of real-time models ● Collect lots of implicit feedback data
- 17. Use Case - eCommerce (cont'd) Detail: ● Clustering - behavior, demogs ● Simple predictors - keywords to products ● Bayesian Bandit - blend the output Requirements: ● Predictions in < 50 ms ● Online learning models ● Occasional batch updates are OK
- 19. When eCommerce #FAILs
- 20. I've only ever bought Cat food...
- 21. ... wait there's more, no Cat food
- 22. Even Amazon can #FAIL
- 24. Use Case - Financial Services Objective - Reduce Fraud How: ● Compute patterns/predictors for individuals ● Cluster individuals and recompute for clusters ● Compute baselines across all data Method: ● Hybrid and Hierarchical Clustering models ● Simple predictors for individuals, clusters & baseline
- 25. Use Case - Financial Services Detail: ● CHEAT!!! ... Cluster to nearest centroid ○ will degrade over time (Hunchback Clusters) ● Use simple metrics to alert (stddev) Requirements: ● Ability to alert/intervene near real-time < 1 second ● Adapt to rapid changes (within baseline & clusters) ● Periodic batch processing to recompute clusters
- 26. Use Case - Financial Services
- 27. Use Case Media
- 28. Use Case - Media Objective - Generating Metadata Why: ● Drive second screen applications ● Create new streams of information for resale How: ● Video / Audio analysis ● Closed Caption or, Subtitle text processing ● Knowledgebase :- People, Places, Products & Things
- 29. Use Case - Media (cont'd) Method: ● Natural Language Processing ○ Named Entity Recognition ○ Topic Extraction & Disambiguation ● Graph databases & algorithms Requirements: ● Responses in < 1 second ● Ability to learn new 'Things' Example of 12,000 entities from our Knowledgebase...
- 33. Summary
- 34. Summary Key points: ● Clear move towards distributed algorithms ● Latency is often more favorable than accuracy ● Trade-offs are dependant on the use-cases Further reading: ● Apache Mahout - http://mahout.apache.org/ ● Storm Project - http://storm-project.net/ ● Data Science London - http://datasciencelondon.org/ ● Machine Learning Meetup - http://bit.ly/w8V8f6
- 35. Almost finished!
- 36. Introducing TUMRA Labs API access to some of our real-time models: ● Probabilistic Demographics Coming Soon: ● Language detection ● Sentiment analysis ● Metadata Generation Free to signup and easy to get started! http://labs.tumra.com/
- 37. Questions? Work Personal tumra.com cotdp.com @tumra @cotdp