SlideShare a Scribd company logo

Walk the Talk: Analyzing the relation between implicit and explicit feedback for preference elicitation

Denis Parra Santander, profile picture
Denis Parra Santander

The document describes a study that analyzed the relationship between implicit and explicit feedback for preference elicitation. The researchers conducted a survey of Last.fm users, collecting demographic data and music listening habits. They also had users rate 100 albums from their listening history. Regression analysis found that implicit feedback (play counts) and recency of listening could predict ratings, but global popularity did not significantly improve predictions. Ongoing work includes incorporating the nested nature of ratings and using alternative evaluation metrics beyond RMSE.

1 of 33
Downloaded 19 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Walk the Talk Analyzing the relation between implicit and explicit feedback for preference elicitation Denis Parra, University of Pittsburgh Dr. Xavier Amatriain, Telefonica R&D UMAP 2011: Girona, Spain 7/12/2011 Parra, Amatriain "Walk the Talk" 1 July 2011
Which is the plan of this presentation? Ah? 7/12/2011 Parra, Amatriain "Walk the Talk" 2
Outline • Introduction (motivation) • The survey on last.fm users – What did we crawl, what did we ask, sampling strategy, results • General analysis of the results • Regression Analysis • Discussion, ongoing work and (some) conclusions 7/12/2011 Parra, Amatriain "Walk the Talk" 3
Which motivation did you have To do this research? 7/12/2011 Parra, Amatriain "Walk the Talk" 4
Introduction (1/2) • Most of recommender system approaches rely on explicit information of the users, but… • Explicit feedback: scarce (people are not especially eager to rate or to provide personal info) • Implicit feedback: Is less scarce, but (Hu et al., 2008) There’s no negative feedback … and if you watch a TV program just once or twice? Noisy … but explicit feedback is also noisy (Amatriain et al., 2009) Preference & Confidence … we aim to map the I.F. to preference (our main goal) Lack of evaluation metrics … if we can map I.F. and E.F., we can have a comparable evaluation 7/12/2011 Parra, Amatriain "Walk the Talk" 5
Introduction (2/2) • Is it possible to map implicit behavior to explicit preference (ratings)? • Which variables better account for the amount of times a user listens to online albums? [Baltrunas & Amatriain CARS ‘09 workshop – RecSys 2009.] • OUR APPROACH: Study with Last.fm users – Part I: demographics and online music consumption – Part II: Rating 100 albums collected from their last.fm user profile 7/12/2011 Parra, Amatriain "Walk the Talk" 6
Which user data did you obtain from last.fm users? 7/12/2011 Parra, Amatriain "Walk the Talk" 7
2.1 Demographic/Consumption Data • Gender • Age • Country • Hours per week spent on internet [int_hrs_per_week] • Hours per week listening to music online [msc_hrs_per_week] • Number of concerts per year [conc_per_year] • Do you read specialized music blogs or magazines? [blogs_mag] • Do you have experience evaluating music online? [rate_music] • How frequently do you buy physical music records? [buy_records] • How frequently do you buy music online? [buy_online] • Do you prefer listening to single tracks, whole albums or either way? [track_or_CD] 7/12/2011 Parra, Amatriain "Walk the Talk" 8
2.2 Albums listened to in last.fm Albums they listened to during last: 7days, 3months, 6months, year, overall For each album in the list we obtained: # user plays (in each period), # of global listeners and # of global plays 7/12/2011 Parra, Amatriain "Walk the Talk" 9
2.3 Survey Part II • For item (album) sampling, we accounted for – Implicit Feedback (IF): playcount for a user on a given album. Changed to scale [1-3], 3 means being more listened to. – Global Popularity (GP): global playcount for all users on a given album [1-3]. Changed to scale [1-3], 3 means being more listened to. – Recentness (R) : time elapsed since user played a given album. Changed to scale [1-3], 3 means being listened to more recently. 7/12/2011 Parra, Amatriain "Walk the Talk" 10
2.4 Survey Screenshots • Requirements: 18 y.o., scrobblings > 5000 7/12/2011 Parra, Amatriain "Walk the Talk" 11
2.5 Survey Part I – Results (1/2) 151 users started 127 users 114 after filtering completed outliers Users from 23 countries 25 users from Spain, 15 from USA, 16 from UK 80% spent 20 or more 50% listened to over 20 hours per week on the hours per week of music internet 9% did not attend music 30% went to 11 or more concerts concerts per year 7/12/2011 Parra, Amatriain "Walk the Talk" 12
2.5 Survey Part I – Results (2/2) 35% read sometimes 20% read them every music magazines or blogs week 50% never or seldom rated music online 45% bought between 1 18% did not buy any and 10 records per year 35% never bought music 8% bought music online at online least once per month 14% preferred listening to 45% preferred listening to single tracks 7/12/2011 albums, ~40% either way 13 Parra, Amatriain "Walk the Talk"
Ok, ok, now… What did you get after analyzing the results? 7/12/2011 Parra, Amatriain "Walk the Talk" 14
3.6 General Analysis - Assumptions • We “expect” strong positive correlation between ratings and implicit feedback • We “expect” some level of positive correlation between ratings and recentness • We (don’t?) expect a significant relation between ratings and global popularity • On demographic data: Just listening to track or album shows a significant effect (using ANOVA) F(2, 62) = 3.949, p = 0.024 7/12/2011 Parra, Amatriain "Walk the Talk" 15
So far so good Are you able to predict ratings with those variables? 7/12/2011 Parra, Amatriain "Walk the Talk" 16
M2: implicit feedback & recentness 4 Regression Analysis M1: implicit feedback M4: M3: implicit Interaction of feedback, implicit recentness, feedback & global recentness popularity • Including Recentness increases R2 in more than 10% [ 1 -> 2] • Including GP increases R2, not much compared to RE + IF [ 1 -> 3] • Not Including GP, but including interaction between IF and RE improves the variance of the DV explained by the regression model. [ 2 -> 4 ] 7/12/2011 Parra, Amatriain "Walk the Talk" 17
4.1 Regression Analysis Model RMSE1 RMSE2 User average 1.5308 1.1051 M1: Implicit feedback 1.4206 1.0402 M2: Implicit feedback + recentness 1.4136 1.034 M3: Implicit feedback + recentness + global popularity 1.4130 1.0338 M4: Interaction of Implicit feedback * recentness 1.4127 1.0332 • We tested conclusions of regression analysis by predicting the score, checking RMSE in 10- fold cross validation. • Results of regression analysis are supported. 7/12/2011 Parra, Amatriain "Walk the Talk" 18
4.2 Regression Analysis – Track or Album Model Tracks Tracks/ Albums Albums User average 1.1833 1.1501 1.1306 M1: Implicit feedback 1.0417 1.0579 1.0257 M2: Implicit feedback + recentness 1.0383 1.0512 1.0169 M3: Implicit feedback + recentness + global 1.0386 1.0507 1.0159 popularity M4: Interaction of Implicit feedback * recentness 1.0384 1.049 1.0159 • Including this variable that seemed to have an effect in the general analysis, helped to improve accuracy of the model 7/12/2011 Parra, Amatriain "Walk the Talk" 19
Is this the end? 7/12/2011 Parra, Amatriain "Walk the Talk" 20
5. Ongoing Work • Incorporate the nested nature of the ratings: they are not independent, so our regression will consider the user as a random factor to group ratings by user (mixed model) • Ratings are not continuous by nature: we will use logistic regression (back up slide for further discussion) • Using raw data (with some transformations) as the value of some predictors (implicit feedback and global popularity) 7/12/2011 Parra, Amatriain "Walk the Talk" 21
6. Conclusions • Using a linear model, Implicit feedback and recentness can help to predict explicit feedback (in the form of ratings) • Global popularity doesn’t show a significant improvement in the prediction task • Our model can help to relate implicit and explicit feedback, helping to evaluate and compare explicit and implicit recommender systems. 7/12/2011 Parra, Amatriain "Walk the Talk" 22
THANKS • For spending your time listening to this talk  • Questions? Denis Parra www.sis.pitt.edu/~dparra/ dap89@pitt.edu 7/12/2011 Parra, Amatriain "Walk the Talk" 23
Survey part I results 7/12/2011 Parra, Amatriain "Walk the Talk" 24
Graphics comparing % of ratings given 2 variables 7/12/2011 Parra, Amatriain "Walk the Talk" 25
Discussion: Evaluating with RMSE • Discussion after working on this paper: Should we continue using RMSE? IS NOT THE SAME MISPREDICTING a rating 2 by 3 than a rating 4 by 5 7/12/2011 Parra, Amatriain "Walk the Talk" 26
Comparison with Study of Duncan Watts • … on evolution of artificial markets, where Watts concludes that people is actually influenced by popularity of items • … but that are important differences with that study: – Watts study was trying to assess music quality, we asked users to avoid judging quality: just if they liked it or not. – In Watts study subjects were presented NEW songs that they HAD TO LISTEN TO in order to judge their quality. 7/12/2011 Parra, Amatriain "Walk the Talk" 27
3.1 Distribution of ratings Average rating: - Considering 0s: 3.206316 - Not considering 0s: 3.611144 7/12/2011 Parra, Amatriain "Walk the Talk" 28
3.2 Implicit Feedback 5 4 3 1 0 7/12/2011 Parra, Amatriain "Walk the Talk" 29
3.3 Recentness 4 5 3 0 1 7/12/2011 Parra, Amatriain "Walk the Talk" 30
3.4 Global Popularity 4 3 5 0 1 7/12/2011 Parra, Amatriain "Walk the Talk" 31
3.5 Effect of Track or CD 5 0 1 7/12/2011 Parra, Amatriain "Walk the Talk" 32
3 General Analysis • Initial assumption: Rating and IF (# playcount) must be strongly correlated. 7/12/2011 Parra, Amatriain "Walk the Talk" 33

More Related Content

PDF
SetFusion Visual Hybrid Recommender - IUI 2014
Denis Parra Santander
 
PPTX
Currents steps to be a researcher and faculty
Denis Parra Santander
 
PDF
Identifying Relevant Messages in a Twitter-based Citizen Channel for Natural ...
Denis Parra Santander
 
PPT
Evaluation of Collaborative Filtering Algorithms for Recommending Articles on...
Denis Parra Santander
 
PPTX
Twitter in Academic Conferences
Denis Parra Santander
 
PPTX
A Hybrid Peer Recommender System for a Online Community Teachers
Denis Parra Santander
 
PDF
LDA on social bookmarking systems
Denis Parra Santander
 
PPTX
Implicit Feedback Recommendation via Implicit-to-Explicit Ordinal Logistic Re...
Denis Parra Santander
 
SetFusion Visual Hybrid Recommender - IUI 2014
Denis Parra Santander
 
Currents steps to be a researcher and faculty
Denis Parra Santander
 
Identifying Relevant Messages in a Twitter-based Citizen Channel for Natural ...
Denis Parra Santander
 
Evaluation of Collaborative Filtering Algorithms for Recommending Articles on...
Denis Parra Santander
 
Twitter in Academic Conferences
Denis Parra Santander
 
A Hybrid Peer Recommender System for a Online Community Teachers
Denis Parra Santander
 
LDA on social bookmarking systems
Denis Parra Santander
 
Implicit Feedback Recommendation via Implicit-to-Explicit Ordinal Logistic Re...
Denis Parra Santander
 

Viewers also liked (6)

PDF
Research on Recommender Systems: Beyond Ratings and Lists
Denis Parra Santander
 
PDF
The Effect of Different Set-based Visualizations on User Exploration of Reco...
Denis Parra Santander
 
PDF
Network Visualization guest lecture at #DataVizQMSS at @Columbia / #SNA at PU...
Denis Parra Santander
 
PDF
Data Fusion for Dealing with the Recommendation Problem
Denis Parra Santander
 
PDF
Interactive Recommender Systems
Denis Parra Santander
 
PDF
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Xavier Amatriain
 
Research on Recommender Systems: Beyond Ratings and Lists
Denis Parra Santander
 
The Effect of Different Set-based Visualizations on User Exploration of Reco...
Denis Parra Santander
 
Network Visualization guest lecture at #DataVizQMSS at @Columbia / #SNA at PU...
Denis Parra Santander
 
Data Fusion for Dealing with the Recommendation Problem
Denis Parra Santander
 
Interactive Recommender Systems
Denis Parra Santander
 
Recommender Systems (Machine Learning Summer School 2014 @ CMU)
Xavier Amatriain
 
Ad

Walk the Talk: Analyzing the relation between implicit and explicit feedback for preference elicitation

  • 1. Walk the Talk Analyzing the relation between implicit and explicit feedback for preference elicitation Denis Parra, University of Pittsburgh Dr. Xavier Amatriain, Telefonica R&D UMAP 2011: Girona, Spain 7/12/2011 Parra, Amatriain "Walk the Talk" 1 July 2011
  • 2. Which is the plan of this presentation? Ah? 7/12/2011 Parra, Amatriain "Walk the Talk" 2
  • 3. Outline • Introduction (motivation) • The survey on last.fm users – What did we crawl, what did we ask, sampling strategy, results • General analysis of the results • Regression Analysis • Discussion, ongoing work and (some) conclusions 7/12/2011 Parra, Amatriain "Walk the Talk" 3
  • 4. Which motivation did you have To do this research? 7/12/2011 Parra, Amatriain "Walk the Talk" 4
  • 5. Introduction (1/2) • Most of recommender system approaches rely on explicit information of the users, but… • Explicit feedback: scarce (people are not especially eager to rate or to provide personal info) • Implicit feedback: Is less scarce, but (Hu et al., 2008) There’s no negative feedback … and if you watch a TV program just once or twice? Noisy … but explicit feedback is also noisy (Amatriain et al., 2009) Preference & Confidence … we aim to map the I.F. to preference (our main goal) Lack of evaluation metrics … if we can map I.F. and E.F., we can have a comparable evaluation 7/12/2011 Parra, Amatriain "Walk the Talk" 5
  • 6. Introduction (2/2) • Is it possible to map implicit behavior to explicit preference (ratings)? • Which variables better account for the amount of times a user listens to online albums? [Baltrunas & Amatriain CARS ‘09 workshop – RecSys 2009.] • OUR APPROACH: Study with Last.fm users – Part I: demographics and online music consumption – Part II: Rating 100 albums collected from their last.fm user profile 7/12/2011 Parra, Amatriain "Walk the Talk" 6
  • 7. Which user data did you obtain from last.fm users? 7/12/2011 Parra, Amatriain "Walk the Talk" 7
  • 8. 2.1 Demographic/Consumption Data • Gender • Age • Country • Hours per week spent on internet [int_hrs_per_week] • Hours per week listening to music online [msc_hrs_per_week] • Number of concerts per year [conc_per_year] • Do you read specialized music blogs or magazines? [blogs_mag] • Do you have experience evaluating music online? [rate_music] • How frequently do you buy physical music records? [buy_records] • How frequently do you buy music online? [buy_online] • Do you prefer listening to single tracks, whole albums or either way? [track_or_CD] 7/12/2011 Parra, Amatriain "Walk the Talk" 8
  • 9. 2.2 Albums listened to in last.fm Albums they listened to during last: 7days, 3months, 6months, year, overall For each album in the list we obtained: # user plays (in each period), # of global listeners and # of global plays 7/12/2011 Parra, Amatriain "Walk the Talk" 9
  • 10. 2.3 Survey Part II • For item (album) sampling, we accounted for – Implicit Feedback (IF): playcount for a user on a given album. Changed to scale [1-3], 3 means being more listened to. – Global Popularity (GP): global playcount for all users on a given album [1-3]. Changed to scale [1-3], 3 means being more listened to. – Recentness (R) : time elapsed since user played a given album. Changed to scale [1-3], 3 means being listened to more recently. 7/12/2011 Parra, Amatriain "Walk the Talk" 10
  • 11. 2.4 Survey Screenshots • Requirements: 18 y.o., scrobblings > 5000 7/12/2011 Parra, Amatriain "Walk the Talk" 11
  • 12. 2.5 Survey Part I – Results (1/2) 151 users started 127 users 114 after filtering completed outliers Users from 23 countries 25 users from Spain, 15 from USA, 16 from UK 80% spent 20 or more 50% listened to over 20 hours per week on the hours per week of music internet 9% did not attend music 30% went to 11 or more concerts concerts per year 7/12/2011 Parra, Amatriain "Walk the Talk" 12
  • 13. 2.5 Survey Part I – Results (2/2) 35% read sometimes 20% read them every music magazines or blogs week 50% never or seldom rated music online 45% bought between 1 18% did not buy any and 10 records per year 35% never bought music 8% bought music online at online least once per month 14% preferred listening to 45% preferred listening to single tracks 7/12/2011 albums, ~40% either way 13 Parra, Amatriain "Walk the Talk"
  • 14. Ok, ok, now… What did you get after analyzing the results? 7/12/2011 Parra, Amatriain "Walk the Talk" 14
  • 15. 3.6 General Analysis - Assumptions • We “expect” strong positive correlation between ratings and implicit feedback • We “expect” some level of positive correlation between ratings and recentness • We (don’t?) expect a significant relation between ratings and global popularity • On demographic data: Just listening to track or album shows a significant effect (using ANOVA) F(2, 62) = 3.949, p = 0.024 7/12/2011 Parra, Amatriain "Walk the Talk" 15
  • 16. So far so good Are you able to predict ratings with those variables? 7/12/2011 Parra, Amatriain "Walk the Talk" 16
  • 17. M2: implicit feedback & recentness 4 Regression Analysis M1: implicit feedback M4: M3: implicit Interaction of feedback, implicit recentness, feedback & global recentness popularity • Including Recentness increases R2 in more than 10% [ 1 -> 2] • Including GP increases R2, not much compared to RE + IF [ 1 -> 3] • Not Including GP, but including interaction between IF and RE improves the variance of the DV explained by the regression model. [ 2 -> 4 ] 7/12/2011 Parra, Amatriain "Walk the Talk" 17
  • 18. 4.1 Regression Analysis Model RMSE1 RMSE2 User average 1.5308 1.1051 M1: Implicit feedback 1.4206 1.0402 M2: Implicit feedback + recentness 1.4136 1.034 M3: Implicit feedback + recentness + global popularity 1.4130 1.0338 M4: Interaction of Implicit feedback * recentness 1.4127 1.0332 • We tested conclusions of regression analysis by predicting the score, checking RMSE in 10- fold cross validation. • Results of regression analysis are supported. 7/12/2011 Parra, Amatriain "Walk the Talk" 18
  • 19. 4.2 Regression Analysis – Track or Album Model Tracks Tracks/ Albums Albums User average 1.1833 1.1501 1.1306 M1: Implicit feedback 1.0417 1.0579 1.0257 M2: Implicit feedback + recentness 1.0383 1.0512 1.0169 M3: Implicit feedback + recentness + global 1.0386 1.0507 1.0159 popularity M4: Interaction of Implicit feedback * recentness 1.0384 1.049 1.0159 • Including this variable that seemed to have an effect in the general analysis, helped to improve accuracy of the model 7/12/2011 Parra, Amatriain "Walk the Talk" 19
  • 20. Is this the end? 7/12/2011 Parra, Amatriain "Walk the Talk" 20
  • 21. 5. Ongoing Work • Incorporate the nested nature of the ratings: they are not independent, so our regression will consider the user as a random factor to group ratings by user (mixed model) • Ratings are not continuous by nature: we will use logistic regression (back up slide for further discussion) • Using raw data (with some transformations) as the value of some predictors (implicit feedback and global popularity) 7/12/2011 Parra, Amatriain "Walk the Talk" 21
  • 22. 6. Conclusions • Using a linear model, Implicit feedback and recentness can help to predict explicit feedback (in the form of ratings) • Global popularity doesn’t show a significant improvement in the prediction task • Our model can help to relate implicit and explicit feedback, helping to evaluate and compare explicit and implicit recommender systems. 7/12/2011 Parra, Amatriain "Walk the Talk" 22
  • 23. THANKS • For spending your time listening to this talk  • Questions? Denis Parra www.sis.pitt.edu/~dparra/ dap89@pitt.edu 7/12/2011 Parra, Amatriain "Walk the Talk" 23
  • 24. Survey part I results 7/12/2011 Parra, Amatriain "Walk the Talk" 24
  • 25. Graphics comparing % of ratings given 2 variables 7/12/2011 Parra, Amatriain "Walk the Talk" 25
  • 26. Discussion: Evaluating with RMSE • Discussion after working on this paper: Should we continue using RMSE? IS NOT THE SAME MISPREDICTING a rating 2 by 3 than a rating 4 by 5 7/12/2011 Parra, Amatriain "Walk the Talk" 26
  • 27. Comparison with Study of Duncan Watts • … on evolution of artificial markets, where Watts concludes that people is actually influenced by popularity of items • … but that are important differences with that study: – Watts study was trying to assess music quality, we asked users to avoid judging quality: just if they liked it or not. – In Watts study subjects were presented NEW songs that they HAD TO LISTEN TO in order to judge their quality. 7/12/2011 Parra, Amatriain "Walk the Talk" 27
  • 28. 3.1 Distribution of ratings Average rating: - Considering 0s: 3.206316 - Not considering 0s: 3.611144 7/12/2011 Parra, Amatriain "Walk the Talk" 28
  • 29. 3.2 Implicit Feedback 5 4 3 1 0 7/12/2011 Parra, Amatriain "Walk the Talk" 29
  • 30. 3.3 Recentness 4 5 3 0 1 7/12/2011 Parra, Amatriain "Walk the Talk" 30
  • 31. 3.4 Global Popularity 4 3 5 0 1 7/12/2011 Parra, Amatriain "Walk the Talk" 31
  • 32. 3.5 Effect of Track or CD 5 0 1 7/12/2011 Parra, Amatriain "Walk the Talk" 32
  • 33. 3 General Analysis • Initial assumption: Rating and IF (# playcount) must be strongly correlated. 7/12/2011 Parra, Amatriain "Walk the Talk" 33