Beyond Collaborative Filtering: Learning to Rank Research Articles
3 likes542 views
This document provides an overview of Elsevier's work using machine learning techniques like collaborative filtering and learning to rank to improve article recommendations for researchers. It discusses using collaborative filtering on browsing logs to initially recommend related articles, then using learning to rank to re-rank those recommendations higher based on features like reputation, topics, citations and user engagement data. Evaluation shows the learning to rank approach improved user engagement by 9-10% over collaborative filtering alone. Future work may explore alternative approaches like graph-based methods or deep learning.
![| 26
Explore/Exploit via Dithering
Slightly shuffle the list of recommendations
• Allows for the exploration of the list
• Gives the impression of freshness
• Reduces some of the bias in LtR training data
!"#$%&'()*+*& = log $012 + 4 0, log 7
where < =
∆ $012
$012
and tipically < ∈ [1.5,2]](https://image.slidesharecdn.com/mayarecsysmeetupldnnov2018latest-181112140653/85/Beyond-Collaborative-Filtering-Learning-to-Rank-Research-Articles-27-320.jpg){kind=icon}
Beyond Collaborative Filtering: Learning to Rank Research Articles
- 1. | 0 Maya Hristakeva (@mayahhf) Beyond Collaborative Filtering: Learning to Rank Research Articles 8th November 2018
- 2. | 1 The Team Data Science, Engineering & Product
- 3. | 2 What do we do?
- 4. | 3
- 5. | 4 Our Users We combine content and data with analytics and technology to help: RESEARCHERS to make new discoveries and have more impact on society CLINICIANS to treat patients better and save more lives NURSES throughout their careers and to help save lives
- 6. | 5 Researcher’s Journey Help me stay up to date Help me showcase my work Help me organise my writing Help me make peer review more rewarding Help me publish faster Help me manage research data Help me with my editorial decisionsHelp me connect with the right people Help me secure funding Help me read and evaluate articles
- 7. | 6 Being the best researcher you can be! • Good researchers are on top of their game • Large amount of research produced • Takes time to get what you need • Help researchers by recommending relevant content
- 8. | 7 Recommenders @ Elsevier
- 9. | 8 Mendeley Suggest – personalized article and people recommenders
- 10. | 9 Science Direct – personalized and related article recommenders
- 11. | 10 Mendeley Funding & Institutional Recommenders
- 12. | 11 Science Direct Related Articles
- 13. | 12 ScienceDirect – related article recommender • Scientific publication database • 15 million articles • 14 million monthly visitors
- 14. | 13 Science Direct V1 Recommender • Goal - Present users with related articles based on the article they are reading • Start simple & iterate - Browsing logs to generate item-to-item CF recommendations - Article content as business logic filtering based on recency, article type
- 15. | 14 Item-based kNN Collaborative Filtering Recommend articles that are similar to the ones you browsed - Similarity is based on article co-occurrences in users’ browsing sessions - “Users who read x also read y” Identify similar articles using cosine similarity: cos $%, $' = )*×), )* × ), Why we use it? - Gives good results - Scales relatively well - Relatively simple to implement
- 16. | 15 Evaluation: Session prediction task • Article browsing logs: • Predict what users would browse next • Time-split evaluation < "#""$%&'(, *+,$-.#'(, *--#""/$0# > Train model Query Ground truth Time, user interactions Test
- 17. | 16 CF & Significance Weighting • Scale down cosine similarity with significance weighting • Preference is given to high co-occurrence neighbors - k – min # sessions in common to get original cosine similarity • Alternative – minimum co-occurrence threshold - Significantly reduces the catalogue coverage score &', &) = min 1, |0'⋂0)| 2 x 345678(&', &))
- 18. | 17 Other CF Improvements • Min/max filters for # articles per user-session & # users-sessions per article • ~ 12 months of browsing logs - gives good coverage - removes cyclical nature of academic year - focuses on more “current” interactions • Bias for recent activity using time decay functions (e.g. exponential) • Using article content as business logic filters for recency and article types
- 19. | 18 Collaborative Filtering in production Recommendations per article IBCF Article views/downloads
- 20. | 19 Can we do any better?
- 21. | 20 A Wealth of Data • Usage Data - Logged-in activity - Alt-metrics, popularity, trending • Social Features - User profiles - Social network - Collaboration groups • > 60 million records: journals, conferences, books, patents … • The most accurate and complete citation & co- author graphs • Reputation metrics for articles, authors and journals • > 15 million full text articles • Article browsing logs • Recommender impression and click logs!!!
- 22. | 21 Learning to Rank (LtR) CF candidates Enriched candidates Re-ranked recs Features LtR model Use CF as candidate selection Enrich with item and user features Re-rank results based on learnt model optimised for CtR
- 23. | 22 LtR Features Reputation & Alt-Metrics Text Topics Temporal Images: wsj, alamy, bookedelic CF similarity score &', &) = min 1, |0'⋂0)| 2 x 345678(&', &)) Citation Network
- 24. | 23 LtR Models • Set of labelled query documents and their associated recommended documents with feature vectors and relevance judgements • Different optimization objectives – point-wise, pair-wise & list-wise • RankLib java-based LtR package - RankNet – pair-wise neural network algorithm - LambdaRank – extension of RankNet optimizing list-wise IR metrics such as NDCG - LambdaMART – list-wise approach combining LambdaRank and MART < "#$%&'()*, %$)'(),*-ℎ/$0-#%$12, %$34)(%$*2 >
- 25. | 24 Recommender Logs LtR requires labelled training data that represents user preferences in relation to the recommendation lists Recommender Logs - Impressions – recs shown to the user - Clicks & conversions – recs the user engaged with - Timestamp – when the event happened - Page-load ID – groups recs that were shown at the same time
- 26. | 25 Training data for LtR Models • Query-recommendations pairs with relevance labels inferred from recommender logs • For each query article - Aggregate the recommended articles across all user sessions - Count # impressions & clicks for each recommendation - Compute graded relevance scores based on CTR
- 27. | 26 Explore/Exploit via Dithering Slightly shuffle the list of recommendations • Allows for the exploration of the list • Gives the impression of freshness • Reduces some of the bias in LtR training data !"#$%&'()*+*& = log $012 + 4 0, log 7 where < = ∆ $012 $012 and tipically < ∈ [1.5,2]
- 28. | 27 Evaluation: Click prediction task • Data: • Rank higher the recommendations users engage with • Time-split evaluation < "#$%&'(&)$*+%,-, &%*(&)$*+%/$)ℎ1%2)#&%3, &%+425%+ > Train model Validation Set Test Set Time, user interactions
- 29. | 28 Results • LtR improved the quality of recommendations - 9-10% improvement in user engagement - Winner is LambdaMART - GBDT with list-wise optimization • LtR increased journal diversity in recommendation lists • LtR promotes recently published articles in the last year • Best ranking model combines usage data with rich structured network and meta data
- 30. | 29 Offline evaluation should match the online challenge • Candidate generation – Collaborative Filtering – session prediction task • Re-ranking candidates – Learning-to-Rank – click prediction task
- 31. | 30 LtR in Production LtR rescoringIBCF Recommendation clicks Training data LtR model Article views/downloads
- 32. | 31 Next Steps & Future Directions
- 33. | 32 Alternative Approaches Graph-based approaches - Random walks for candidate generation Deep Learning - Learn more complex features for LtR - Neural embeddings for candidate generation - Hybrid systems for ranking
- 34. | 33 Evaluation – correcting for bias & confounding • Algorithm confounding - How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. Allison J. B. Chaney, Brandon M. Stewart, and Barbara E. Engelhardt (RecSys '18). • Explore/exploit – multi-armed bandits - Explore, exploit, and explain: personalizing explainable recommendations with bandits. James McInerney, et al. (RecSys ‘18). • Counterfactuals - Counterfactual reasoning and learning systems: The example of computational advertising. Bottou, Léon, et al. (JMLR 2013).
- 35. | 34 Qualitative & Quantitative Evaluation https://github.com/jeanigarcia/recsys2018-evaluation-tutorial
- 36. | 35 Challenges
- 37. | 36 Recommender Team Publications Hristakeva, M., Kershaw, D., Pettit, B., Vargas, S., & Jack, K. (2019). Academic recommendations: The Mendeley case. In Collaborative Recommendations: Algorithms, Practical Challenges and Applications. Pettit, B., Hristakeva, M., Kershaw, D. & Jack, K. (2018). Learning to Rank Research Articles: A case study of collaborative filtering and learning to rank in Science Direct. Hristakeva, M., Kershaw, D., Rossetti, M., Knoth, P., Pettit, B., Vargas, S., & Jack, K. (2017). Building recommender systems for scholarly information. WSDM2017. Rossetti, M., Vargas, S., Pettit, B., Kershaw, D., Hristakeva, M., & Jack, K. (2017). Effectively identifying users’ research interests for scholarly reference management and discovery. WSDM2017. Vargas, S., Hristakeva, M., & Jack, K. (2016). Mendeley: Recommendations for Researchers. RecSys ’16
- 38. | 37 References From RankNet to LambdaRank to LambdaMART: An Overview (2010). Christopher J. C. Burges. On Application of Learning to Rank for E-Commerce Search by Shubhra Kanti Karmaker Santu, Parikshit Sondhi, and ChengXiang Zhai (SIGIR 2017). Recommender Systems Handbook (2010). Francesco Ricci, Lior Rokach, Bracha Shapira, and Paul B. Kantor. Practical Machine Learning: Innovations in Recommendation (2014). Ted Dunning and Ellen Friedman. O'Reilly Media, Inc. Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time by Chantat Eksombatchai, Pranav Jindal, Jerry Zitao Liu, Yuchen Liu, Rahul Sharma, Charles Sugnet, Mark Ulrich, and Jure Leskovec (WWW 2018). Getting Deep Recommenders Fit: Bloom Embeddings for Sparse Binary Input/Output Networks by Joan Serrà and Alexandros Karatzoglou (RecSys 2017)
- 39. We're hiring, come speak to us! https://www.elsevier.com/about/careers/technology-careers