Creating an end-to-end Recommender System with Apache Spark and Elasticsearch - Nick Pentreath & Jean-François Puget
- 1. Spark Technology Center Oct / 27 / 16 Creating an end-to-end Recommender System with Apache Spark and Elasticsearch Jean-François Puget Nick Pentreath
- 2. Spark Technology Center § @JFPuget § Distinguished Engineer, IBM Machine Learning & Optimization § @MLnick § Principal Engineer, IBM Spark Technology Center § Apache Spark PMC About
- 3. Spark Technology Center § Recommender systems & the machine learning workflow § Data modelling for recommender systems § Why Spark & Elasticsearch? § Spark ML for collaborative filtering § Deploying & scoring recommender models § Demo Agenda
- 4. Spark Technology Center Recommender Systems & the ML Workflow
- 6. Spark Technology Center The Machine Learning Workflow Perception Data ??? Machine Learning ??? $$$
- 7. Spark Technology Center The Machine Learning Workflow Reality Data • Historical • Streaming Ingest Data Processing • Feature transformation & engineering Model Training • Model selection & evaluation Deploy • Pipelines, not just models • Versioning Live System • Predict given new data • Monitoring & live evaluation Feedback Loop Spark DataFrames Spark ML Various ??? Stream (Kafka) Missing piece!
- 8. Spark Technology Center The Machine Learning Workflow Recommender Version Data Ingest Data Processing • Aggregation • Handle implicit data Model Training • ALS • Ranking-style evaluation Deploy • Model size & complexity Live System •User & item recommendations •Monitoring, filters Feedback => another Event Type Spark DataFrames Spark ML Elasticsearch • User & Item Metadata • Events Elasticsearch Stream (Kafka)
- 9. Spark Technology Center Data Modeling for Recommender Systems
- 10. Spark Technology Center Data modelUser and Item Metadata ! !
- 11. Spark Technology Center System RequirementsUser and Item Metadata ! ! Filtering & Grouping Business Rules
- 12. Spark Technology Center User interactions Implicit preference data • Page view • eCommerce - cart, purchase • Media – preview, watch, listen Intent data • Search query Anatomy of a User Event Explicit preference data • Rating • Review Social network interactions • Like • Share • Follow User Interactions ! ! ! ! ! ! ! !
- 13. Spark Technology Center Data modelAnatomy of a User Event ! ! ! !! ! !
- 14. Spark Technology Center How to handle implicit feedback?Anatomy of a User Event ! ! ! !! ! ! !
- 15. Spark Technology Center Why Spark & Elasticsearch?
- 16. Spark Technology Center DataFrames § Events & metadata are “lightly structured” data § Suited to DataFrames § Pluggable external data source support Spark ML § Spark ML pipelines § Scalable ALS algorithm, supporting implicit feedback & NMF § Cross-validation § Custom transformers & algorithms Why Spark?
- 17. Spark Technology Center Storage § Native JSON § Scalable § Good support for time-series / event data § Kibana for data visualisation § Integration with Spark DataFrames Scoring § Full-text search § Filtering § Aggregations (grouping) § Search ~== recommendation (more later) Why Elasticsearch?
- 18. Spark Technology Center Spark ML for Collaborative Filtering
- 19. Spark Technology Center Matrix FactorizationCollaborative Filtering 3 4 1 5 2 1 3 2 1 ! ! −1.1 3.2 4.3 0.2 1.4 3.1 2.5 0.3 2.3 4.3 −2.4 0.5 3.6 0.3 1.2 0.2 1.7 2.3 1.9 0.4 0.8 1.5 −1.2 0.3 −0.4 2.1 0.6 2.7 0.8 1.4 ! !
- 20. Spark Technology Center PredictionCollaborative Filtering 3 4 1 5 2 1 3 2 1 ! ! −1.1 3.2 4.3 0.2 1.4 3.1 2.5 0.3 2.3 4.3 −2.4 0.5 3.6 0.3 1.2 0.2 1.7 2.3 1.9 0.4 0.8 1.5 −1.2 0.3 −0.4 2.1 0.6 2.7 0.8 1.4 ! !
- 21. Spark Technology Center Loading DataAlternating Least Squares
- 22. Spark Technology Center Implicit Preference DataAlternating Least Squares
- 23. Spark Technology Center Deploying & Scoring Recommendation Models
- 24. Spark Technology Center Full-text Search & SimilarityPrelude: Search “cat videos” ! !cat videos 0 0 ⋯ 0 1 ⋯ 0 1 ⋯ 1 1 ⋯ 1 1 ⋯ 0 0 ⋯ 1 0 ⋯ 0 1 ⋯ Sort results 0 1 ⋯ 1 0 ⋯ Scoring RankingAnalysis Term vectors Similarity
- 25. Spark Technology Center Can we use the same machinery?Recommendation ! 0 0 ⋯ 0 1 ⋯ 0 1 ⋯ 1 1 ⋯ 1 1 ⋯ 0 0 ⋯ 1 0 ⋯ 0 1 ⋯ Sort results 1.2 ⋯ −0.2 0.3 Dot product & cosine similarity … the same as we need for recommendations! Scoring RankingAnalysis Term vectors ! !!! SimilarityUser (or item) vector ?
- 26. Spark Technology Center Delimited Payload FilterElasticsearch Term Vectors Raw vector 1.2 ⋯ −0.2 0.3 Term vector with payloads 0|1.2 ⋯ 3|-0.2 4|0.3 Custom analyzer
- 27. Spark Technology Center Custom scoring function • Native script (Java), compiled for speed • Scoring function computes dot product by: § For each document vector index (“term”), retrieve payload § score += payload * query(i) • Normalize with query vector norm and document vector norm for cosine similarity (“similar items”) Elasticsearch Scoring
- 28. Spark Technology Center Can we use the same machinery?Recommendation ! Sort results 1.2 ⋯ −0.2 0.3 Scoring RankingAnalysis Term vectors !! Custom scoring function !! Delimited payload filter −1.1 1.3 ⋯ 0.4 1.2 −0.2 ⋯ 0.3 0.5 0.7 ⋯ −1.3 0.9 1.4 ⋯ −0.8 ! User (or item) vector
- 29. Spark Technology Center We get search engine functionality for free!Elasticsearch Scoring
- 30. Spark Technology Center Deploying to ElasticsearchAlternating Least Squares
- 31. Spark Technology Center Monitoring & Feedback
- 33. Spark Technology Center Elasticsearch Elasticsearch Spark Integration Spark ML ALS for Collaborative Filtering Collaborative Filtering for Implicit Feedback Datasets Elasticsearch Term Vectors & Payloads Delimited Payload Filter Vector Scoring Plugin Kibana References