Workshop on Designing Human-Centric MIR Systems
- 1. Supporting user control for music recommendations Dr. Nava Tintarev Assistant Professor TU Delft/Delft Data Science/Delft Design for Values
- 2. Research area & Domains Program chair: ACM IUI (Cagliari, 2020) Senior reviewer: ACM IUI, Recsys, UMAP Reviewer, a.o., IJCAI-ECAI, CHI, ACM TiiS Music News Travel Education
- 4. What is a recommender system? Systems that make personalized recommendations of goods, services, and people (Kautz) • User identifies one or more objects as being of interest • The recommender system suggests other objects that are similar (infers liking) • Ranking and filtering algorithm • Ranks the options, filters out lower ranking options 1 2 3 4
- 7. Explanations Read this you must, because… Tintarev & Masthoff, 2012
- 8. Purpose Description Transparency Explain how the system works Effectiveness Help users make good decisions Trust Increase users' confidence in the system Persuasiveness Convince users to try or buy Satisfaction Increase the ease of use or enjoyment Scrutability Allow users to tell the system it is wrong Efficiency Help users make decisions faster Tintarev and Masthoff. 2007
- 9. Timeline 2006| 2008 | 2010 | 2012 | 2014 | 2016 | 2018 Adaptive text explanations Better explanations? • Vizualizations • Interactive interfaces Factors that influence expl. effectiveness? • Situational Characteristics • Personal Characteristics
- 11. Personal Characteristics? • Working memory (Conati2014, Tintarev2016, Tocker2012, Velex2005) • Expertise (Kobsa2001, Shah2011, Tocker2012) Situational Characteristics? • Perceptions of diversity (Ferwerda2017, Kang2016, Willemsen2011) • Group recommendations (Najafian2018, Kapcak2018)
- 13. Research questions • How do personal characteristics influence the impact of… • RQ1: user controls in terms of diversity and acceptance? • RQ2: visualizations in terms of diversity and acceptance? • RQ3: visualizations + user controls in terms of diversity and acceptance? 13 Katrien Verbert Associate Professor Yucheng Jin PhD researcher Jin et al 2018: UMAP and Recsys, UMUAI 2019
- 14. Study procedure Tutorial of study Pre-study questionnaire: • Demographics • Experience • Visual Memory (VM) Capacity • Musical Sophistication (MS) Task • Pick scenario • Create a play- list • Explore and rate all songs in the final playlist Post-study questionnaire: • perceived quality • perceived accuracy • perceived diversity • Satisfaction • Effectiveness • choice difficulty
- 15. Procedure Platform • Spotify API generate recommendations • Based on up to five favourite artists. • 14 musical attributes in order to describe musical preference Dependent variables • Perceived diversity: self-reported measure based on questionnaire • Recommendation acceptance: measured by percentage of liked songs in the play-list 15
- 16. Personal characteristics (Co-variates) Musical sophistication (MS) Measurement of the ability to engage with music in a flexible, effective and nuanced way (Müllensiefen et al., 2014) Measured using the Goldsmiths Musical Sophistication Index (Gold-MSI) Visual memory (VM) the ability to recall visual patterns (Tintarev and Mastoff, 2016) Measured by Corsi block-tapping test
- 17. User-centered factors 17 Exp.1 and Exp.3 used framework of Knijnenburg et al. 2012. Perceived accuracy: participants' perceived accuracy of the recommended songs according to their preference. Perceived diversity: the similarity among the recommended songs. •Several songs in the list of recommended songs were very different from each other. •The list of recommended songs covered many genres •Most songs were from the same type Exp. 2 used the ResQue framework (Pu2011) The songs recommended to me are of various kinds. The songs recommended to me are similar to each other.
- 18. Hypotheses • H1: The UI setting (user control, visualization, or both) has a significant effect on recommendation acceptance. • H2: The UI setting (user control, visualization, or both) has a significant effect on perceived diversity. • H3: Visual memory (VM) has a significant effect on recommendation acceptance. • H4: Visual memory (VM) has a significant effect on perceived diversity. • H5: Musical sophistication (MS) has a significant effect on recommendation acceptance. • H6: Musical sophistication (MS) has a significant effect on perceived diversity. 18
- 19. Participants 19
- 20. User profile (PRO) Algorithm parameters (PAR) Recommendations (REC) Experiment1:Controls
- 21. CFA - validity questions Quality Accuracy Effectiveness Choice difficulty Diversity (low AVE value) Trust (low AVE value) Satisfaction (modification index) The fit of our SEM model X(2, 98) = 257.410 p <.001 RMSEA= 0.083 CFI = 0.980 TLI = 0.968
- 22. “Sweetspot” of control • Two-way interaction: does not necessarily corr. w higher cognitive load. Adding an additional control component to PAR increases corr w. acceptance. PRO*PAR has lower cognitive load than PRO and PAR • Three-way interaction: correlates w. acceptance, and does not correlate with sig. higher cognitive load. Increase interaction times and accuracy • High MS correlated w higher quality, which corr. with higher rec. acceptance 2 2
- 23. UI Design 23 Experiment 2: Visualizations
- 24. Exp. 3: Control + Vis. 24
- 25. Summary • Exp1: Acceptance (MS) • Exp2: Diversity (MS, VS) • Exp3: • Acceptance (MS) • Diversity (MS, VS)
- 26. Summary • User control: Musical sophistication interacted w the acceptance of recommendations for user controls. • Visualization: • Musical sophistication also interacted w. perceived diversity for visualizations, but only for one of the studied visualizations. • Visual memory interacted with visualizations for perceived diversity, but only for one of the studied visualizations.
- 28. ContextPlay (Jin, 2019) • RQ1: How does adding control of contextual characteristics influence user perceptions of the system? • RQ2: How do different contextual characteristics (mood, weather, time of day, and social) influence user perceptions Location Activity
- 30. ContextPlay (results) • H1: The control of context positively influence user perception of system. (Accept) - RQ1 • H2: The contextual characteristics will influence user perception of system. (Accept) - RQ2 • E.g. Mood positively influences perceived quality (p<.05) and diversity (p<.05), which further positively influences the perceived effectiveness. • Both Weather (positively) and location (outdoor, negatively) influence effectiveness (p<.05).
- 31. The percentage of users who controlled context In general, mood is th in all scenarios, and w as much as mood in th scenario. The most manipulated components are the liked artists and tracks in four scenarios. • Mood is the most modified – all scenarios • Weather – as much for outdoor + relaxing scenario Controlling context
- 32. X The percentage of users who controlled each user profile age of users who controlled context anipulated components are ists and tracks in four g scenarios seem to engage to modify their profile rios will influences user n control. (Accept) - RQ3 • Most modified artists and tracks (all scenarios) • Relaxing scenarios (most modifications) • H3: Scenarios influence user requirements on control Controlling user profile
- 33. Summary
- 34. What’s next? • ENSURE - ExplaiNing SeqUences in Recommendations for groups with divergent preferences in music and tourism. (2018-2022) • XAINL - European Training Network on Explainable AI using Natural Language • Interactions to mitigate human biases • FAIRView (Google Digital News Initiative) produce explainable video summaries (2019-2020): • to assess the quality of the video summaries in terms of how well they represent different aspects of the original video • to assess the potential for misinformation or bias • Representing diverse views for polarized topics online (2019-2023, co- funded IBM) • Devising Metrics for Assessing Echo Chambers, Incivility, and Intolerance (2019-2021, funded by Twitter)
- 35. EPSILON LAB – EXPLANATIONS Post-docs Oana Inel Emily Sullivan Mesut Kaya (research engineer DDS) PhDs Shabnam Najafian Yucheng Jin (KU Leuven) Tim Draws Twitter (TBA) Multi-stakeholder recs (TBA) nl4xai.eu (TBA) PhD and Postdoc openings planned!
- 37. References (1) • Conati, C., Carenini, G., Hoque, E., Steichen, B., and Toker, D. (2014). “Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making,” in Eurographics Conference on Visualization (EuroVis). • Ferwerda, B., Graus, M. P., Vall, A., Tkalcic, M., & Schedl, M. (2017, April). How item discovery enabled by diversity leads to increased recommendation list attractiveness. In Proceedings of the Symposium on Applied Computing (pp. 1693-1696). ACM. • Jin, Y., Htun, N.N., Tintarev, N., and Verbert, K.. "Contextplay: user control for context- aware music recommendation". In ACM Conference on User Modeling and Adaptation and Personalization (UMAP). 2019. • Jin, Y., Tintarev, N., & Verbert, K. (2018, September). Effects of personal characteristics on music recommender systems with different levels of controllability. In Proceedings of the 12th ACM Conference on Recommender Systems (pp. 13-21). ACM. • Jin, Y., Tintarev, N., & Verbert, K. (2018, July). Effects of individual traits on diversity- aware music recommender user interfaces. In Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization (pp. 291-299). ACM. • Jin, Y., Htun, N.N., Tintarev, N., and Verbert, K (2019). "Effects of personal characteristics in control-oriented user interfaces for music recommender systems". User Modeling and User-Adapted Interaction, 2019. • Kang, B., Tintarev, N., Höllerer, T., & O’Donovan, J. (2016). What am I not seeing? An interactive approach to social content discovery in microblogs. In International Conference on Social Informatics (pp. 279-294). Springer, Cham.
- 38. References (2) • Kapcak, Ö., Spagnoli, S., Robbemond, V., Vadali, S., Najafian, S. and N., Tintarev (2018) TourExplain: A Crowdsourcing Pipeline for Generating Explanations for Groups of Tourists. ACM RecSys Workshop on Recommenders in Tourism. • Knijnenburg, B. P., Willemsen, M. C., Gantner, Z., Soncu, H., & Newell, C. (2012). Explaining the user experience of recommender systems. User Modeling and User- Adapted Interaction, 22(4-5), 441-504. • Kobsa, A. (2001). “An empirical comparison of three commercial information visualization systems,” in IEEE Symposium on Information Visualization (InfoVis) (San Diego, CA). • Müllensiefen, D., Gingras, B., Musil, J., & Stewart, L. (2014). The musicality of non- musicians: an index for assessing musical sophistication in the general population. PloS one, 9(2), e89642. • Najafian, S., & Tintarev, N. (2018). Generating Consensus Explanations for Group Recommendations: an exploratory study. In Adjunct Publication of the 26th Conference on User Modeling, Adaptation and Personalization (pp. 245-250). ACM. • Pu, P., Chen, L., & Hu, R. (2011). A user-centric evaluation framework for recommender systems. In Proceedings of the fifth ACM conference on Recommender systems (pp. 157- 164). ACM. • Shah, P., and Freedman, E. G. (2011). Bar and line graph comprehension: an interaction of top-down and bottom-up processes. Top. Cogn. Sci. 3, 560–578.
- 39. References (3) • N. Tintarev & J. Masthoff, Evaluating the effectiveness of explanations for recommender systems: methodological issues and empirical studies on the impact of personalization UMUAI 2012 • (Tintarev2018a) Tintarev, N., Rostami, S., & Smyth, B. (2018). Knowing the unknown: visualising consumption blind-spots in recommender systems. In ACM Symposium On Applied Computing (SAC). • (Tintarev2018b) Tintarev, N., Sullivan, E., Guldin, D., Qiu, S., & Odjik, D. (2018). Same, Same, but Different: Algorithmic Diversification of Viewpoints in News. In Adjunct Publication of the 26th Conference on User Modeling, Adaptation and Personalization (pp. 7-13). • Toker, D., Conati, C., Carenini, G., and Haraty, M. (2012). “Towards adaptive information visualization: on the influence of user characteristics,” in User Modeling, Adaptation, and Personalization. • Velez, M. C., Silver, D., and Tremaine, M. (2005). “Understanding visualization through spatial ability differences,” in VIS 05. IEEE Visualization, 2005, 511– 518. • Willemsen, M. C., Knijnenburg, B. P., Graus, M. P., Velter-Bremmers, L. C., & Fu, K. (2011). Using latent features diversification to reduce choice difficulty in recommendation lists. RecSys, 11, 14-20.
- 40. Acceptance 40 H1: The UI setting has a significant effect on recommendation acceptance (Cannot accept)
- 41. Perceived diversity 41 Full Control vs Control+ComBub (M=5.43, SD=1.07, p = .005) Full Control vs Control+SimBub (M=5.22, SD=1.20, p = .035)
- 42. Personal characteristics 42 Correlation results on acceptance and perceived diversity (* denotes statistical significance at the 0.05 level, i.e., p value< 0.05)
- 43. Discussion • Adding visualization to Full Control seems to increase diversity significantly. • Control + SimBub • Positive correlation between MS and two dependent variables (acceptance, perceived diversity) • Control + ComBub • Positive correlation between personal characteristics (MS, VM) and the perceived diversity in ComBub • MS also has a positive effect on acceptance • users with high MS are good at tuning the recommender to find high quality recommendations 43
- 44. 44
Editor's Notes
- #12: positive effect on satisfaction if they enriched a user’s tastes (Ferwerda, 2017)] perceived diversity led to decreased difficulty to make a choice (Willemsen et al., 2011)
- #23: Main effects: from REC to PRO to PAR → higher cognitive load ?
- #31: Mood positively influences perceived quality (p<.05) and diversity (p<.05), which further positively influences the perceived effectiveness. Mood SSA Effectiveness (EXP) Num. of listened songs (INT) Both Weather (positively) and location (negatively) influence effectiveness (p<.05).
- #32: Mood positively influences perceived quality (p<.05) and diversity (p<.05), which further positively influences the perceived effectiveness. Mood SSA Effectiveness (EXP) Num. of listened songs (INT) Both Weather (positively) and location (negatively) influence effectiveness (p<.05).
- #33: Mood positively influences perceived quality (p<.05) and diversity (p<.05), which further positively influences the perceived effectiveness. Mood SSA Effectiveness (EXP) Num. of listened songs (INT) Both Weather (positively) and location (negatively) influence effectiveness (p<.05).
- #35: FairNews (NWA) investigates to what extent algorithms can and may go into filtering information for the purpose of "fairness" (2018-2019)