Interactive Recommender Systems

Interactive Recommender Systems:
Bridging the gap between predictive
algorithms and interactive user interfaces
DenisParra,Ph.D.InformationSciences
AssistantProfessor,CS Department
Schoolof Engineering
PontificiaUniversidadCatólicadeChile
UFMG,March29th 2017

Outline
• Brief Personal Introduction
• Computer Science at PUC Chile
• Projects at SocVis Lab
• Overview of Recommender Systems
• Interactive Recommender Systems
• Summary & Current & Future Work
March 29th, 2017 D.Parra ~ UFMG– Invited Talk 2

1 slide Geography Class: Chile
• One third of the 16 million Chileans lives in Santiago,
the Capital
• But Chile is a looong country (4.000 Km), in the north
is hot and dry, in the south (Patagonia) is very cold.
Very Hot!
Very Cold!
My hometown!
Valdivia
Santiago, PUC Chile

Personal Introduction 1/3
• B.Eng. and Engineering in Informatics from
Universidad Austral de Chile (2004), Valdivia, Chile
• Ph.D. in Information Sciences at University of
Pittsburgh (2008-2013), Pittsburgh, PA, USA

• In 2009 I did an internship at Trinity College
Dublin, with researcher Alexander Troussov (IBM)
• In 2010 I did another internship at Telefonica I+D,
Barcelona, with Xavier Amatrian (now VP Quora)

• 2013: Moved back to Santiago, Chile
• Department of CS, School of Engineering, PUC.

DCC, Engineering, PUC Chile
• DCC: Departamento de Ciencia de la Computación
• Programs: BEng, Engineering title, Master, PhD
• Research Areas:
– Databases and Semantic Web
– Information Technologies
– Machine Learning and Computer Vision (GRIMA)
– Software Engineering
– Educational Technologies,MOOCs
http://dcc.ing.puc.cl

Academic activities (2017)
• Research topics: Recommender
Systems/Personalization, Visualization, SNA.
• Teaching: Data Mining, Recommender Systems,
Information Visualization, SNA.
• Leading the Social Computing and Visualization
(SocVis) Lab.

SocVis Lab
http://www.socvis.cl

People, Publications, News (ND)

Projects at SocVis
• Mood-based music artists recommendation
– Collaboration with J. O’Donovan (UCSB)
– Student: Raimundo Herrera
• IR on evidence-based Medicine
– Help doctors on answering clinical questions
– Student: I. Donoso, collaboration Epistemonikos
• Artwork Recommendation
– Collaboration with online artwork store UGallery
– Students: P. Messina & V. Dominguez

Recommender Systems Class
• Recommender Systems at PUC Chile
http://web.ing.puc.cl/~dparra/classes/recsys-2016-2/

INTRODUCTIONTORECSYS
Recommender Systems
* Danboard (Danbo): Amazon’s cardboard robot, in these slides it represents a
recommender system
*

Recommender Systems (RecSys)
Systems that help (groups of) people to find relevant items in
a crowded item or information space(MacNee et al. 2006)

Why do we care about RecSys?
• RecSys have gained popularity due to several
domains & applications that require people to make
decisions among a large set of items.

A lil’ bit of History
• First recommender systems were built at the
beginning of 90’s (Tapestry, GroupLens, Ringo)
• Online contests, such as the Netflix prize, grew the
attention on recommender systems beyond
Computer Science
(2006-2009)

The Recommendation Problem
• The most popular way of presenting the
recommendation problem is rating prediction:
• How good is my prediction?
Item 1 Item 2 … Item m
User 1 1 5 4
User 2 5 1 ?
…
User n 2 5 ?
Predict!

Recommendation Methods
• Without covering all possible methods, the two
most typical classifications on recommender
algorithms are
Classification 1 Classification 2
- Collaborative Filtering
- Content-based Filtering
- Hybrid
- Memory-based
- Model-based

Collaborative Filtering (User-based KNN)
• Step 1: Finding Similar Users (Pearson Corr.)
5
4
4
1
2
1
5
4
4
1
2
5
Active
user
User_1
User_2
User_3
active user
user_1
user_2
user_3

• Step 1: Finding Similar Users (Pearson Corr.)
5
4
4
1
2
1
5
4
4
1
2
5
Active
user
User_1
User_2
User_3
∑∑
∑
⊂⊂
⊂
−−
−−
=
nunu
nu
CRi nniCRi uui
CRi nniuui
rrrr
rrrr
nuSim
,,
,
22
)()(
))((
),(
active user
user_1 0.4472136
user_2 0.49236596
user_3 -0.91520863

• Step 2: Ranking the items to recommend
5
4
4
2
1
5
4
4
Active
user
User_1
User_2
2
3
4
2
Item 1
Item 2

• Step 2: Ranking the items to recommend
5
4
4
2
1
5
4
4
Active
user
User_1
User_2
∑
∑
⊂
⊂
−⋅
+=
)(
)(
),(
)(),(
),(
uneighborsn
uneighborsn nni
u
nuuserSim
rrnuuserSim
riupred2
3
4
2
Item 1
Item 2
Item 3

Pros/Cons of CF
PROS:
• Very simple to implement
• Content-agnostic
• Compared to other techniques such as content-
based, is more accurate. There is also the Item KNN.
CONS:
• Sparsity
• Cold-start
• New Item

Content-Based Filtering
• Can be traced back to techniques from IR, where
the User Profile represents a query.
user_profile = {w_1, w_2, …., w_3} using TF-IDF, weighting
Doc_1 = {w_1, w_2, …., w_3}
Doc_2 = {w_1, w_2, …., w_3}
Doc_3 = {w_1, w_2, …., w_3}
Doc_n = {w_1, w_2, …., w_3}
5
4
5

PROS/CONS of Content-Based Filtering
PROS:
• New items can be matched without previous
feedback
• It can exploit also techniques such as LSA or LDA
• It can use semantic data (ConceptNet, WordNet,
etc.)
CONS:
• Less accurate than collaborative filtering
• Tends to overspecialization

Hybridization
• Combine previous methods to overcome their
weaknesses (Burke, 2002)

C2. Model/Memory Classification
• Memory-based methods use the whole dataset in
training and prediction. User and Item-based CF are
examples.
• Model-based methods build a model during training
and only use this model during prediction. This
makes prediction performance way faster and
scalable

Model-based: Matrix Factorization
Latent vector of the item
Latent vector of the user
SVD ~
Singular Value
Decomposition

PROS/CONS of MF and latent factors model
PROS:
• So far, state-of-the-art in terms of accuracy (these
methods won the Netflix Prize)
• Performance-wise, the best option nowadays: slow
at training time O((m+n)3) compared to correlation
O(m2n), but linear at prediction time O(m+n)
CONS:
• Recommendations are obscure: How to explain that
certain “latent factors” produced the
recommendation?

Other paradigms and techniques
• Recommendation as a graph problem:
– Model the problem as diffusion or link prediction
– Personalized PageRank (Kamvar et al, 2010), (Santos et
al 2016)
• Recommendation as a ranking problem:
– Rather than predicting ratings, predict a Top-N list
– Learning-to-rank approaches developed in the IR
community
– Karatzoglou et al. (2013), Shi et al. (2014), Macedo et al.
(2015)

(Important) RecSys Topics
Not Covered in this Presentation
• Learning to rank
• Graph-based methods
• Context-aware recommenders
• Recommendation problem as next-item in sequence
• User-centric evaluation frameworks
• Multiarmed Bandits
• Reinforcement Learning
• ... You need to take Professor Santos’ course J

Rethinking the Recommendation Problem
• User feedback is scarce: need for exploiting different
sources of user preference and context

• Ratings are scarce: need for exploiting other sources
of user preference
• User-centric recommendation takes the problem
beyond ratings and ranked lists: evaluate user
engagement and satisfaction, not only RMSE/MAP

• Ratings are scarce: need for exploiting other sources
of user preference
• User-centric recommendation takes the problem
beyond ratings and ranked lists: evaluate user
engagement and satisfaction, not only RMSE/MAP
• Several other dimensions to consider in the
evaluation: novelty of the results, diversity,
coverage (user and catalog), trust
• Study de effect of interface characteristics:
controllability, transparency, explainability.

My Take on RecSys Research (2009 ~)

My Work on RecSys
• In my research I have contributed to RecSys by:
– Utilizing other sourcesof user preference(Social Tags)
– Exploiting implicit feedback for recommendation and for
mapping explicit feedback
– Studying interactive interfaces:the effect of visualizations and
user interactionon user satisfaction, perceptionof trust and
accuracy.
• Nowadays: Focus on interactive exploratory interfaces for
recommender systems

This is not only My work J
• Dr. Peter Brusilovsky
University of Pittsburgh, PA, USA
• Dr. Alexander Troussov
IBM Dublin and TCD, Ireland
• Dr. Xavier Amatriain
TID / Netflix /Quora
• Dr. Christoff Trattner
NTNU, Norway
• Dr. Katrien Verbert
KU Leuven, Belgium
• Dr. Leandro Balby-Marinho
UFCG, Brasil

VISUALIZATION+USER
CONTROLLABILITY
Part of this work with Katrien Verbert

Human Factors in RecSys
• Transparency and Explainability: Konstan et al (2000),
Tintarev and Mastoff (2010)
• Frameworks to evaluate RecSys user studies: ResQue (Pu et al
, 2010), Knijnenburg et al (2012)
• Controllability and Inspectability: O’Donovan (2008),
Knijnenburg et al (2010, 2012),Hijikata (2012), Ekstrand et al
(2015)
• Visualization andInterfaces: O’Donovan (2008 - ..),
Verbert et al (2013), Parra et al (2014), Loepp et al (2014,
2017),

Visualization & User Controllability
• Motivation: Can user controllability and
explainability improve user engagement and
satisfaction with a recommender system?
• Specific research question: How intersections of
contexts of relevance (of recommendation
algorithms) might be better represented for user
experience with the recommender?

Traditional RecSys Interface
MovieLens: example of traditional recommender list

Explanations and Control
Options of
User Control
Explainability
Recommendations of books
GoodReads: Book recommender system

PeerChooser (2008) Controllability in CF
O’Donovan et al. “PeerChooser: Visual Interactive Recommendation” (2008)

SmallWorlds: Expanded Explainability
Gretarsson et al. “SmallWorlds: Visualizing social recommendations” (2010)

TasteWeights: Hybrid Control and Inspect
Bostandjev et al. “TasteWeights: A Visual Interactive Hybrid Recommender System” (2012)
Controllability:
Sliders that let users
control the
importance of
preferences and
contexts
Inspectability: lines
that connect
recommended items
with contexts and user
preferences

IUI 2017
• Loepp et al. (2017)

More Details? Check our survey
He, C., Parra, D., & Verbert, K. (2016). Interactive recommender
systems: a survey of the state of the art and future research
challenges and opportunities. Expert Systems with Applications, 56, 9-27.

Visualization & User Controllability
• Motivation: Can user controllability and
explainability improve user engagement and
satisfaction with a recommender system?
• Specific research question: How overlapping
contexts of relevance (of recommendation
algorithms) might be better represented for user
experience with the recommender?
• Our scenario: Conference articles

Research Platform
• The studies were conducted using Conference
Navigator, a Conference Support System
• Our goal was recommending conference talks
Program Proceedings Author List Recommendations
http://halley.exp.sis.pitt.edu/cn3/

TalkExplorer – IUI 2013
• Adaptation of Aduna Visualization to CN
• Main research question: Does fusion (intersection) of
contexts of relevance improve user experience?

TalkExplorer - I
Entities
Tags, Recommender Agents,
Users

TalkExplorer - II
Recommender
Recommender
Cluster
with
intersecti
on of
entities Cluster (of
talks)
associated to
only one
entity
• Canvas Area: Intersections of Different Entities
User

TalkExplorer - III
Items
Talks explored by the
user

Our Assumptions
• Items which are relevant in more that
one aspect could be more valuable to the
users
• Displaying multiple aspects of relevance
visually is important for the users in the
process of item’s exploration

TalkExplorer Studies I & II
• Study I
– Controlled Experiment:Users were asked to discover
relevant talksby exploring the three types of entities: tags,
recommenderagents and users.
– Conducted at Hypertext and UMAP 2012 (21 users)
– Subjects familiar with Visualizations and Recsys
• Study II
– Field Study: Users were left free to explore the interface.
– Conducted at LAK 2012 and ECTEL 2013 (18 users)
– Subjects familiar with visualizations, but not much with
RecSys

Evaluation: Intersections & Effectiveness
• What do we call an “Intersection”?
• We used #explorations on intersections and their
effectiveness, defined as:
Effectiveness =

Results of Studies I & II
• Effectiveness increases
with intersections of
more entities
• Effectiveness wasn’t
affected in the field
study (study 2)
• … but exploration
distribution was
affected

More Details About TalkExplorer
• Verbert, K., Parra, D., Brusilovsky, P., & Duval, E. (2013).
Visualizing recommendationsto support exploration,
transparencyand controllability. In Proceedingsof the 2013
internationalconference on Intelligent user interfaces(pp. 351-362).
ACM.
• Verbert, K., Parra, D., & Brusilovsky, P. (2016). Agents Vs.Users:
Visual Recommendationof ResearchTalks with Multiple
Dimension of Relevance. ACM Transactionson Interactive
Intelligent Systems (TiiS), 6(2), 11.

SETFUSION:VENNDIAGRAMFOR
USER-CONTROLLABLEINTERFACE

SetFusion – IUI 2014

SetFusion I
Traditional
Ranked List
Paperssorted by
Relevance.
It combines3
recommendation
approaches.

SetFusion - II
Sliders
Allow the user to control the importance of
each data source or recommendation method
Interactive Venn Diagram
Allows the user to inspect and to filter papers
recommended. Actionsavailable:
- Filter item list by clicking on an area
- Highlight a paper by mouse-over on a circle
- Scroll to paper by clicking on a circle
- Indicate bookmarkedpapers

Study : iConference
• A laboratory within-subjectsstudy. 40 subjects.
• In Preferenceelicitation phase, people did not have limit of
papers. Under RecSys interfaces, minimum limit was 15.
• In bookmarking, subjects could pick items relevant to a)
themselves, b) themselves and others, and c) only to others.
$12/hour
Avg: 1.5 hours

Study : Population and General Stats
Non-controllable Controllable
# Total bookmarks 638 625
# Average bookmarks/user 15.95 15.63
# Average rating 2.48±0.089 2.46±0.076
Gender Female: 17 Male: 23
Age 31.75±6.5
Native Speaker Yes: 10 No: 30
Subject Occupation Information Sc. (16), Library Sc.(9), Comp. Sc. (6), Telecomm (3), (+6)
PCA
15 questions
on
pre-questionnaire
4 Factors (User
Characteristics)
• Expertise in domain
• Engaged with iSchools
• Trusting Propensity
• Experience w/RecSys
Dropped
• Experience w/CN

Study 2: Results (1)
Variables Comment
User Engagement
Significant Talks explored, clicks (nbr. actions) , time spent on task All significantly higher
in controllable interface
User Experience
Significant MAP Significantly higher in
controllable interface
User Characteristics
Significant Trusting prop.:increases use of Venn diagram and MAP
Native speaker: Decreases time spent on task
Gender: Being male increases use of sliders
Age: Each additional year decreases use of sliders
Trusting propensity
confirms results of
previous studies

Rating per method – Effect of Visuals

Gender Differences on SetFusion?

Study : Results (2)
Post-session surveys Controllable No-Controllable
Understandability 4.05±0.09*** 2.95±0.16
Satisfaction with interface 4.28±0.09*** 3.4±0.16
Confidence of not missing relevant talks 3.9±0.11*** 3.13±0.15
Intention: I would use it again 4.23±0.09*** 3.45±0.15
Intention: I would recommend system to colleagues 4.28±0.09*** 3.48±0.16
Venn diagram visualization was useful to identify talks
recommended by a specific or by a combination of
recommendation methods.
4.35±0.11 --
Venn diagram visualization supported explainability
4.08±0.13 --
Satisfaction due to ability to control 4.05±0.12 --
Perception of Control with Sliders 4.03±0.13 --

Study : Results (3)
Non-control Controllable Both None
Which interface did you prefer? 0 36 4 0
Non-control Controllable None Both
Which interface would you suggest
to implement permanently in CN?
1 33 1 5
“I like the Venn diagram especially because most papers I was interested in fell in the same
intersections, so it was pretty easy to find and bookmark”
“I thought the controllable one adds unnecessary complication if the list is not very long”
“I prefer the sliders (over Venn diagram) because I have used a system before to control
search results with a similar widget, so it was more familiar to me.”

Study Takeaways
• User Engagement: Controllable interface significantly
drives more user engagement (objective and subjective
metrics)
• User Experience: Controllable interface improves
user experience by allowing user to interactively
control ranking (MAP) and improving explainability.
• User characteristics: Trusting propensity affects
positively engagement and experience, engagement
with iSchools shows the opposite. Males have a
tendency to prefer sliders over Venn diagram to control
and filter.

More Details on SetFusion?
• Effect of other variables: gender, age, experience
with in the domain, familiarity with the system
• Check our paper in the IJHCS “User-controllable
Personalization: A Case Study with SetFusion”:
Controlled Laboratory study with SetFusion versus
traditional ranked list

Study 2 – UMAP 2013
• Field Study: let users freely explore the interface
- ~50% (50 users) tried the
SetFusion recommender
- 28% (14 users) bookmarked at
least one paper
- Users explored in average 14.9
talks and bookmarked 7.36
talks in average.
A AB ABC AC B BC C
15 7 9 26 18 4 17
16% 7% 9% 27% 19% 4% 18%
Distribution of bookmarks per method or combination of methods

Hybrid RecSys: Visualizing Intersections
Clustermap Venn diagram
• Clustermap vs. Venn Diagram

TalkExplorer vs. SetFusion
• Comparing distributions of explorations
In studies 1 and 2 over
TalkExplorer we observed an
important change in the
distribution of explorations.

TalkExplorer vs. SetFusion
• Comparing distributions of explorations
Comparing the field studies:
- In TalkExplorer, 84% of
the explorationsover
intersectionswere
performed over clusters of
1 item
- In SetFusion, was only
52%, compared to 48%
(18% + 30%) of multiple
intersections, diff. not
statistically significant

Summary & Conclusions
• We showed that intersections of several contexts of
relevance help to discover relevant items
• The visual paradigm used can have a strong effect on
user behavior: we need to keep working on visual
representations that promote exploration without
increasing the cognitive load over the users

Limitations & Future Work
• Apply our approach to other domains (fusion of
data sources or recommendation algorithms)
• For SetFusion, find alternatives to scale the
approach to more than 3 sets, potential alternatives:
– Clustering and
– Radial sets
• Consider other factors that interact with the user
satisfaction:
– Controllability by itself vs. minimum level of accuracy

Current Work on Interfaces
• MoodPlay
– With Ivana Andjelkovic & John O’Donovan (UCSB)
Andjelkovic, I., Parra, D., & O'Donovan, J. (2016, July). Moodplay:
Interactive Mood-based Music Discovery and Recommendation.
In Proceedings of the 2016 Conference on User Modeling Adaptation and
Personalization (pp. 275-279). ACM.

MoodPlay
• https://www.youtube.com/watch?v=eEdo32oOmcE

Emotion Models
• Modelo de emociones de Russel
(1980)
• GEMS (2008)

Moods and Music: the GEMS model

System Architecture

Hybrid Recommendation Approach

User Study
• Conducted on Mechanical Turk, 4 conditions

Interactions

Interaction Stats

Diversity Consumption

User Prior Mood and Artist Mood

Post-Study Survey
Accuracy

Post-Study Survey
Diversity

Post-Study Survey
Confusing Interface

Post-Study Survey
Easy to use

CONCLUSIONS (&CURRENT)&
FUTUREWORK

Challenges in Interactive RecSys
• Objectives
• Controllability
• Context-aware recommendations
• Privacy
• Visualization Techniques
• Interaction Techniques
• Conversation Interfaces
• Evaluation Methodology

Future Work
• Opportunities for using new devices (Sensors on
Stmartphones, EEG)
• Although new devices can capture a lot of new types
of data, there is still a lot to be done with data we
already produce but we haven’t consumed (user logs
on social web sites, etc.)
3/29/17 D. Parra, FuturePDtalk, UMAP 2016 95

MoodPlay in the Chilean news
Moodplay as therapy?

Moodplay as therapy?
• S. Koelsch. A neuroscientific perspective on music
therapy. Annals of the New York Academy of Sciences,
1169(1):374–384, 2009.
• Music can help on modulate certain mental states.

Previous work: MIT Mood Meter
• http://moodmeter.media.mit.edu/

Input Data: from Social Networks?
• Michelle Zhou’s personality profile

Visual emotion detection
• https://github.com/auduno/clmtrackr

Using EEG (BCI)
EMOTIV
http://emotiv.com/epoc/
NEUROSKY
http://neurosky.com/bi
osensors/

Heatmaps to Moodplay

EpistAid
• Epistemonikos: Evidence-based Medicine
• Physicians answer clinical questions

EpistAid 2
• Process of building evidence matrices is really slow

EpistAid: IUI to support physicians
• Study 1: Relevance Feedback to find missing papers
faster, off-line evaluation
• Study 2: Study with physicians at PUC

Interactive Recommender Systems

More Related Content

What's hot (20)

Viewers also liked (16)

Similar to Interactive Recommender Systems (20)

More from Denis Parra Santander (14)

Recently uploaded (20)

Interactive Recommender Systems