Investigating Serendipity in Recommender Systems Based on Real User Feedback
- 1. Investigating Serendipity in Recommender Systems Based on Real User Feedback Denis Kotkov, University of Jyväskylä, Finland Joseph A. Konstan, University of Minnesota, USA Qian Zhao, University of Minnesota, USA Jari Veijalainen, University of Jyväskylä, Finland
- 2. Summary • In this study, we • summarize definitions of serendipity in recommender systems • show that serendipity is valuable for users
- 3. Outline • Introduction • Definitions • Motivation • Experiment • Results • Conclusion • Dataset
- 4. Recommender systems Recommender systems are software tools that suggest items of use to users (Ricci et al., 2015) Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 5. From accuracy to serendipity • Accurate recommendations: • Too safe or boring • Familiar • Users would consume them anyway • How about serendipitous recommendations? Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 6. What is serendipity? Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 7. Dictionary The faculty of making fortunate discoveries by accident (https://www.thefreedictionary.com/serendipity) Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 8. Dictionary The faculty of making fortunate discoveries by accident (https://www.thefreedictionary.com/serendipity) Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 9. Dictionary The faculty of making fortunate discoveries by accident (https://www.thefreedictionary.com/serendipity) Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 10. Dictionary The faculty of making fortunate discoveries by accident (https://www.thefreedictionary.com/serendipity) Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 11. Scientific sources First, a serendipitous item should be not yet discovered and not be expected by the user; secondly, the item should also be interesting, relevant and useful to the user. (Ge et al., 2010) Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 12. Scientific sources Serendipity represents the “unusualness” or “surprise” of recommendations. (Zhang et al., 2012) Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 13. What is serendipity in recommender systems? • Serendipity components (Kotkov et al., 2016) • Relevance • Novelty • Unexpectedness Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 14. What is serendipity in recommender systems? • Serendipity components • Relevance – the user likes or is interested in the item • Novelty • Unexpectedness Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 15. What is serendipity in recommender systems? • Serendipity components • Relevance • Novelty • Strict novelty – the user has never heard about this item • Motivational novelty – the user was motivated to consume this item by the recommender system • Unexpectedness Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 16. What is serendipity in recommender systems? • Serendipity components • Relevance • Novelty • Unexpectedness • Unexpectedness (relevant) – the user does not expect to like an item before consuming it • Unexpectedness (find) – the user does not think they would find the item on their own • Unexpectedness (implicit) – the item is dissimilar to those the user usually consumes • Unexpectedness (recommend) – the user does not expect to this item to be recommended Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 17. What is serendipity in recommender systems? Name Relevance Strict novelty Motivational novelty Unexpectedness (relevant) Unexpectedness (find) Unexpectednes s (implicit) Unexpectedness (recommend) Strict serendipity (relevant) + + + Strict serendipity (find) + + + Strict serendipity (implicit) + + + Strict serendipity (recommend) + + + Motivational serendipity (relevant) + + + Motivational serendipity (find) + + + Motivational serendipity (implicit) + + + Motivational serendipity (recommend) + + + Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 18. Which variation is the most valuable? Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 19. Why serendipitous items? • According to the literature review, serendipity • broadens user preferences (Herlocker et al., 2004; Zhang et al., 2012) • increases user satisfaction (Lu et al., 2012; Murakami et al., 2007) Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 20. Not enough evidence • Takeaways: • Users prefer using serendipity-oriented algorithms over accuracy-oriented ones • Users give low ratings to serendipitous items on average • Focus is on the algorithm • Sample size: 21 users Zhang, Y. C., Séaghdha, D. Ó., Quercia, D., & Jambor, T. (2012, February). Auralist: introducing serendipity into music recommendation. In Proceedings of the fifth ACM international conference on Web search and data mining (pp. 13-22). ACM. Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 21. Is serendipity valuable for users? Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 22. Research questions 1. What are the effects of variations of novelty and unexpectedness on preference broadening and user satisfaction? 2. What are the effects of serendipity variations on preference broadening and user satisfaction? 3. What are the effective features for detecting serendipitous items? 4. How rare are serendipitous items? Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 23. Research questions 1. What are the effects of variations of novelty and unexpectedness on preference broadening and user satisfaction? 2. What are the effects of serendipity variations on preference broadening and user satisfaction? 3. What are the effective features for detecting serendipitous items? 4. How rare are serendipitous items? Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 24. MovieLens Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 25. Survey • Selection criteria for movies: • recently watched • rated with at least 3.5 stars (relevant) • unpopular Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 26. Survey • We picked 5 movies that are the least popular and rated with at least 3.5 stars during the three months November December January February March April The beginning of the survey The user joined MovieLens At least 5 ratings >= 3.5 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 27. Survey Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 28. Survey Statement Entity The first time I heard of this movie was when MovieLens suggested it to me strict novelty MovieLens influenced my decision to watch this movie motivational novelty I expected to enjoy this movie before watching it for the first time unexpectedness (relevant) This is the type of movie I would not normally discover on my own; I need a recommender system like MovieLens to find movies like this one unexpectedness (find) This movie is different (e.g., in style, genre, topic) from the movies I usually watch unexpectedness (implicit) I was (or, would have been) surprised that MovieLens picked this movie to recommend to me unexpectedness (recommend) Watching this movie broadened my preferences. Now I am interested in a wider selection of movies preference broadening I am glad I watched this movie user satisfaction Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 29. Survey Concept User-movie pairs Users All 2146 475 Strictly serendipitous (relevant) 77 (3%) 61 Strictly serendipitous (find) 181 (8%) 119 Strictly serendipitous (implicit) 115 (5%) 80 Strictly serendipitous (recommend) 63 (3%) 50 Motivationally serendipitous (relevant) 91 (4%) 64 Motivationally serendipitous (find) 163 (7%) 101 Motivationally serendipitous (implicit) 128 (5%) 88 Motivationally serendipitous (recommend) 71 (3%) 49 Serendipitous 302 (14%) 173 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 30. Effects of novelty and unexpectedness • Coefficients of ordinal regression models (12 models) • metric ~ component Component Preference broadening User satisfaction Strict novelty 0.74* 0.12 Motivational novelty 0.78* 0.3 Unexpectedness (relevant) 0.59* -0.91* Unexpectedness (find) 2.11* 0.19 Unexpectedness (implicit) 1.86* -0.03 Unexpectedness (recommend) 1.08* -0.45 Significance codes: “*” < 0.0005 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 31. Effects of novelty and unexpectedness • Coefficients of ordinal regression models (12 models) • metric ~ component Component Preference broadening User satisfaction Strict novelty 0.74* 0.12 Motivational novelty 0.78* 0.3 Unexpectedness (relevant) 0.59* -0.91* Unexpectedness (find) 2.11* 0.19 Unexpectedness (implicit) 1.86* -0.03 Unexpectedness (recommend) 1.08* -0.45 Significance codes: “*” < 0.0005 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 32. Effects of novelty and unexpectedness • Coefficients of ordinal regression models (12 models) • metric ~ component Component Preference broadening User satisfaction Strict novelty 0.74* 0.12 Motivational novelty 0.78* 0.3 Unexpectedness (relevant) 0.59* -0.91* Unexpectedness (find) 2.11* 0.19 Unexpectedness (implicit) 1.86* -0.03 Unexpectedness (recommend) 1.08* -0.45 Significance codes: “*” < 0.0005 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 33. Effects of novelty and unexpectedness • Coefficients of ordinal regression models (12 models) • metric ~ component Component Preference broadening User satisfaction Strict novelty 0.74* 0.12 Motivational novelty 0.78* 0.3 Unexpectedness (relevant) 0.59* -0.91* Unexpectedness (find) 2.11* 0.19 Unexpectedness (implicit) 1.86* -0.03 Unexpectedness (recommend) 1.08* -0.45 Significance codes: “*” < 0.0005 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 34. Effects of novelty and unexpectedness • Results for novelty variations were not statistically significant • Coefficients of ordinal regression models (6 regression models) • Preference broadening ~ a variation of unexpectedness Component Unexpectedness (relevant) Unexpectedness (find) Unexpectedness (implicit) Unexpectedness (find) 0.82* Unexpectedness (implicit) 0.63* -0.13 Unexpectedness (recommend) 0.40 -0.39 -0.18 Significance codes: “*” < 0.0005 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 35. Effects of novelty and unexpectedness • Results for novelty variations were not statistically significant • Coefficients of ordinal regression models (6 regression models) • User satisfaction ~ a variation of unexpectedness Component Unexpectedness (relevant) Unexpectedness (find) Unexpectedness (implicit) Unexpectedness (find) 0.63* Unexpectedness (implicit) 0.46* -0.14* Unexpectedness (recommend) 0.14 -0.45 -0.32 Significance codes: “*” < 0.0005 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 36. Effects of novelty and unexpectedness • Results for novelty variations were not statistically significant • Coefficients of ordinal regression models (6 regression models) • User satisfaction ~ a variation of unexpectedness Component Unexpectedness (relevant) Unexpectedness (find) Unexpectedness (implicit) Unexpectedness (find) 0.63* Unexpectedness (implicit) 0.46* -0.14* Unexpectedness (recommend) 0.14 -0.45 -0.32 Significance codes: “*” < 0.0005 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 37. Effects of novelty and unexpectedness • If your goal is to improve preference broadening: • All variations of novelty and unexpectedness are good • Unexpectedness (find) is the best choice • Unexpectedness (relevant) is the worst choice Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 38. Effects of serendipity • Results regarding user satisfaction were not statistically significant • Coefficients of ordinal regression models (8 regression models) • Preference broadening ~ a variation serendipity Serendipity variation Preference broadening Strict serendipity (relevant) 0.97* Strict serendipity (find) 1.47* Strict serendipity (implicit) 1.58* Strict serendipity (recommend) 1.6* Motivational serendipity (relevant) 0.6 Motivational serendipity (find) 1.66* Motivational serendipity (implicit) 1.35* Motivational serendipity (recommend) 1.3* Significance codes: “*” < 0.0005 Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 39. Effects of serendipity • Results regarding preference broadening were not statistically significant • In terms of user satisfaction, • Motivational serendipity (find) > strict serendipity (implicit) > motivational serendipity (relevant) • Strict serendipity (recommend) > motivational serendipity (recommend) Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 40. Takeaways • Preference broadening – yes • User satisfaction – unclear • Serendipity variations are different • Which serendipity variation to choose? • Motivational serendipity (find) seems to be the best choice • Motivational and strict serendipity (relevant) seem to be the worst choice Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 41. Dataset • User answers • Recommendations generated by MovieLens • Tag genome • Movie data (such as title, genres, cast) • 10M user ratings https://grouplens.org/datasets/serendipity-2018/ Introduction Definitions Motivation Experiment Results Conclusion Dataset
- 42. References Ge, M., Delgado-Battenfeld, C., & Jannach, D. (2010, September). Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proceedings of the fourth ACM conference on Recommender systems (pp. 257-260). ACM. Herlocker, J. L., Konstan, J. A., Terveen, L. G., & Riedl, J. T. (2004). Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS), 22(1), 5-53. Kotkov, D., Wang, S., & Veijalainen, J. (2016). A survey of serendipity in recommender systems. Knowledge- Based Systems, 111, 180-192. Lu, Q., Chen, T., Zhang, W., Yang, D., & Yu, Y. (2012, December). Serendipitous personalized ranking for top- n recommendation. In Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology-Volume 01 (pp. 258-265). IEEE Computer Society. Murakami, T., Mori, K., & Orihara, R. (2007, June). Metrics for evaluating the serendipity of recommendation lists. In Annual Conference of the Japanese Society for Artificial Intelligence (pp. 40-46). Springer, Berlin, Heidelberg. Ricci, F., Rokach, L., & Shapira, B. (2015). Recommender systems: introduction and challenges. In Recommender systems handbook (pp. 1-34). Springer, Boston, MA. Zhang, Y. C., Séaghdha, D. Ó., Quercia, D., & Jambor, T. (2012, February). Auralist: introducing serendipity into music recommendation. In Proceedings of the fifth ACM international conference on Web search and data mining (pp. 13-22). ACM.
- 43. Questions?