![<A, P>
<A, N>
Anchor
cross_entropy(
[<A,P>, <A,N>, ...],
[1 , 0 , …]
)
Positive
Negative
Dot Product Loss Function](https://image.slidesharecdn.com/deeplearningrecommendersystems-datatalks-210323170901/85/Deep-Learning-Recommender-Systems-20-320.jpg)
Deep Learning Recommender Systems
- 1. Deep Learning Recommender Systems from Prototype to Production DataTalks.Club 23.03.21
- 2. Outline ● About us ● Modern Recommender Systems ● Deep Learning Recommender Systems ● Neural Item Embeddings ● Similarity Search ● Proving value through Experimentation ● From POC to PRD ● Summary
- 3. About us ILIA IVANOV Data Scientist ilia.ivanov@olx.com CRISTIAN MARTINEZ Lead Data Scientist cristian.martinez@olx.com
- 6. Almost all modern platforms that we use daily have some kind of recommender system
- 7. Amazon Amazon researchers found that using neural networks to generate movie recommendations worked much better when they sorted the input data chronologically and used it to predict future movie preferences over a short (one- to two-week) period. Source: https://www.amazon.science/the-history-of-amazons-recommendation-algorithm
- 8. Deep Learning Recommender Systems (DLRS)
- 9. Content-based Deep content-based music recommendation (Oord et al, 2013) ● Convolutional neural nets ● Key ideas: ○ Extract latent representations of songs from audio signals ○ Train CNN network to generate embeddings of songs ○ The input of the CNN is a time-frequency representation of the audio
- 10. Neural Item Embeddings E-commerce in Your Inbox (Grbovic et al, 2015) ● Inspired by word2vec architecture ● Key ideas: ○ Generate embeddings of products in a word2vec fashion ○ “Words” are products and “sentences” are purchase sequences of a user ○ Learn product embeddings using skip-gram
- 11. Wide & Deep Wide & Deep Learning for Recommender Systems (Cheng et al, 2016) ● Wide and deep feed-forward architectures ● Key ideas: ○ Jointly train two neural networks. ○ Deep neural network trained with embeddings. ○ (Wide) Linear model with feature transformations for generic recommender systems with sparse inputs.
- 12. Session-based Session-based recommendations with RNNs (Hidasi et al, 2016) ● Recurrent neural network architectures ● Key ideas: ○ Train using sequence of items in a session ○ Predict: next item in the session ○ Pairwise (ranking) loss functions ■ Bayesian personalised ranking (BPR)
- 13. Graph-based Graph Convolutional Neural Networks for Web-Scale Recommender Systems (Ying et al, 2018) ● GCN architectures ● Key ideas: ○ Combines efficient random walks and graph convolutions to generate embeddings of nodes (e.g. items) ○ Improve quality of item embedding by extracting Information from its neighbor nodes.
- 14. Benefits of using DNNs ● Extract deep features directly from content ○ e.g. CNNs for images, LSTMs for text ● Allow us to combine information extracted from different input sources ● Generate more accurate representations of users and items ● Allow us to use custom loss functions
- 16. ITEM EMBEDDINGS
- 17. 2018 Deep neural network marketplace recommenders in online experiments Combine content-based features with user behaviors to solve the cold start challenge of collaborative filtering. 2019 How we use item2vec to recommend similar products Avito’s item2vec implementation. +30% contacts from i2i recommendations. +20% contacts from personal recommendations. 2015 E-commerce in Your Inbox: Product Recommendations at Scale Use word2vec algorithm to learn embedding representations of products. 2016 Meta-Prod2Vec: Product Embeddings Using Side-Information for Recommendation Prod2vec + meta-data about the item.
- 18. text location params category item neural network LSTM layer word embeddings Concatenate Embedding layer Embedding layer Dense layer Dropout Dense (ReLU) Dense (Tanh) Embedding space
- 19. Similarity Assumption time Pair 1 Pair 2 Pair 3
- 20. <A, P> <A, N> Anchor cross_entropy( [<A,P>, <A,N>, ...], [1 , 0 , …] ) Positive Negative Dot Product Loss Function
- 22. References ● E-commerce in Your Inbox: Product Recommendations at Scale (Grbovic et al, 2016) ● Meta-Prod2Vec: Product Embeddings Using Side-Information for Recommendation (Vasile et al, 2016) ● Deep neural network marketplace recommenders in online experiments (Eide et al, 2018) ● Как мы используем item2vec для рекомендаций похожих товаров Avito’s item2vec (Russian language)
- 24. Search Problem: find the nearest neighbors among millions of item embeddings Solution: approximate nearest neighbor (ANN) Implementations: ● FAISS (FB) ● Annoy (Spotify) ● NMSLIB ● Open Distro for ES
- 25. Experimentation
- 26. Off-line Model Quality Development Speed loss function generic feature encoders automated feature selection domain-specific data (& model) data cleaning
- 27. Pre-online Internal testing framework to ● compare recommendations ● debug
- 28. On-line -18% +0.6% +16% +9% Iterate over 1 model in the corresponding domain Roll-out to more domains & improve incrementally Contacts effect over time
- 29. From POC to PRD
- 30. Automatization AWS SageMaker AWS Batch Airflow
- 32. Summary
- 33. Lessons Learned ● DL + RS = ❤ ● Start simple + iterate ● Learn from your users (A/B-testing)
- 34. Q&A Thank you!