![PREDICTION PROBLEM
UserID Song 1 Song2 Score ... delta
contempt
delta
valence
...
... ... ... ... ... ... ... ...
14 34 123 -2 ... 0.34 -0.11 ...
... ... ... ... ... ... ... ...
• Regression problem: Score in [-2,+2]
• Classification problem: Score element of {-1,0,1}](https://image.slidesharecdn.com/tkalciciui2019-190320005045/85/Prediction-of-Music-Pairwise-Preferences-from-Facial-Expressions-16-320.jpg)
![PREDICTION PROBLEM
• Regression problem: Score in [-2,+2]
• Classification problem: Score element of {-1,0,1}
• Baseline: listening time
• ML: Random Forest, Gradient Boosting
• 5x random sampling 60/40](https://image.slidesharecdn.com/tkalciciui2019-190320005045/85/Prediction-of-Music-Pairwise-Preferences-from-Facial-Expressions-17-320.jpg)
![RESULTS
• Regression problem:
Score in [-2,+2]
• Classification problem:
Score element of {-1,0,1}](https://image.slidesharecdn.com/tkalciciui2019-190320005045/85/Prediction-of-Music-Pairwise-Preferences-from-Facial-Expressions-18-320.jpg)
![RESULTS
• Regression problem:
Score in [-2,+2]
• Classification problem:
Score element of {-1,0,1}](https://image.slidesharecdn.com/tkalciciui2019-190320005045/85/Prediction-of-Music-Pairwise-Preferences-from-Facial-Expressions-19-320.jpg)
Prediction of Music Pairwise Preferences from Facial Expressions
- 1. PREDICTION OF MUSIC PAIRWISE PREFERENCES FROM FACIAL EXPRESSIONS MARKOTKALČIČ1, NIMA MALEKI2, MATEVŽ PESEK3, MEHDI ELAHI1, FRANCESCO RICCI1, MATIJA MAROLT3 1 FREE UNIVERSITY OF BOZEN-BOLZANO, ITALY 2VODAFONE ITALY 3 UNIVERSITY OF LJUBLJANA, SLOVENIA
- 2. BACKGROUND 1. RecSys Preference Elicitation 2. Pairwise Preferences 3. Emotional Response THIS WORK
- 3. BACKGROUND -1: RECSYS PREFERENCE ELICITATION • Explicit • Intrusive • Accurate • Ratings,Thumb Up/Down • Implicit • Unobtrusive • Inaccurate • Listening time, Play-count
- 4. BACKGROUND-1: RECSYS PREFERENCE ELICITATION • Explicit • Intrusive • Accurate • Ratings,Thumb Up/Down • Implicit • Unobtrusive • Inaccurate -> Less inaccurate • Listening time, Play-count
- 5. BACKGROUND-2: PAIRWISE PREFERENCES • Single judgment • Benchmark? • Context-dependent • Pairwise judgment • Comparable items
- 6. BACKGROUND-2: PAIRWISE PREFERENCES • Single Judgment • Benchmark? • Context-dependent • Pairwise Judgment • Comparable Items
- 7. BACKGROUND-3: EMOTIONAL RESPONSE • Emotions are triggered (cf Mood) • Music induces emotions • Various models • Dimensional (Valence,Arousal, Dominance) • Basic Emotions (Fear, Joy,Anger, Disgust, Surprise, Sadness) • Emotions can be measured from facial expressions
- 8. RESEARCH QUESTION Can we infer implicitly pairwise music preferences of a user from the facial expressions during listening to songs? SONG 1 SONG 2 PREDICTION MODEL
- 9. PROPOSED SOLUTION • IUI 2019 • Data Acquisition (User Study) • Prediction of Pairwise Preferences (ML) • Future work • Evaluation in Pairwise RecSys
- 10. DATA ACQUISITION • Scenario: • Music • Fixed context: music for work (cognitively demanding task) • Items: • 200 songs • 15s snippets • Unknown (familiarity debiased) • Subjects • 75 • 29.8 yr (SD=9.5) • 26F, 49M
- 11. USER STUDY
- 12. USER STUDY
- 15. ACQUIRED DATA UserID Song 1 Song2 Score ... delta contempt delta valence ... ... ... ... ... ... ... ... ... 14 34 123 -2 ... 0.34 -0.11 ... ... ... ... ... ... ... ... ... • UserID, Song1, Song2, Score • Raw facial expressions • Facial expression features per song pair • polynomial fitting 2nd order
- 16. PREDICTION PROBLEM UserID Song 1 Song2 Score ... delta contempt delta valence ... ... ... ... ... ... ... ... ... 14 34 123 -2 ... 0.34 -0.11 ... ... ... ... ... ... ... ... ... • Regression problem: Score in [-2,+2] • Classification problem: Score element of {-1,0,1}
- 17. PREDICTION PROBLEM • Regression problem: Score in [-2,+2] • Classification problem: Score element of {-1,0,1} • Baseline: listening time • ML: Random Forest, Gradient Boosting • 5x random sampling 60/40
- 18. RESULTS • Regression problem: Score in [-2,+2] • Classification problem: Score element of {-1,0,1}
- 19. RESULTS • Regression problem: Score in [-2,+2] • Classification problem: Score element of {-1,0,1}
- 20. RESULTS • Differences between user clusters • RMSE
- 21. CONCLUSION • New approach for implicit acquisition of preferences • Facial expressions explain the variance in preferences • Important features • contempt • valence • joy • sadness • Privacy issues • Future work: evaluation in RecSys