Replication of Recommender Systems Research

Replication of Recommender
Systems Research
Alejandro Bellogín
Universidad Autónoma de Madrid
@abellogin
Alan Said
University of Skövde
@alansaid
ACM RecSys Summer School 2017

Who are we?
Alejandro Bellogín
•  Lecturer (~Asst. Prof) @ Universidad
Autónoma de Madrid, Spain
•  PhD @ UAM, 2012
•  Research on
–  Evaluation
–  Similarity metrics
–  Replication & reproducibility
Alan Said
•  Lecturer (~Asst. Prof) @ University
of Skövde, Sweden
•  PhD @ TU Berlin, 2013
•  Research on
–  Evaluation
–  Replication & reproducibility
–  Health
2ACM RecSys Summer School 201725-Aug-17

Outline
•  Motivation [10 minutes]
•  Replication and reproducibility
•  Replication in Recommender Systems
•  Demo
•  Conclusions and Wrap-up [10 minutes]
•  Questions [10 minutes]
25-Aug-17 ACM RecSys Summer School 2017 3

Motivation
In RecSys, we ﬁnd inconsistent results,
for the “same”
–  Dataset
–  Algorithm
–  Evaluation metric
Movielens 1M
[Cremonesi et al, 2010]
Movielens 100k
[Gorla et al, 2013]
Movielens 1M
[Yin et al, 2012]
Movielens 100k, SVD
[Jambor & Wang, 2010]
425-Aug-17 ACM RecSys Summer School 2017

Motivation
for the “same”
–  Dataset
–  Algorithm
Movielens 1M
Movielens 100k
[Gorla et al, 2013]
Movielens 1M
[Yin et al, 2012]
Movielens 100k, SVD
So what?

Motivation
A proper evaluation culture allows the ﬁeld to advance
… or at least, identify when there is a problem!

Motivation
for the “same”
–  Dataset
–  Algorithm
Movielens 1M
Movielens 100k
[Gorla et al, 2013]
Movielens 1M
[Yin et al, 2012]
Movielens 100k, SVD

Motivation
for the “same”
–  Dataset
–  Algorithm
Movielens 1M
Movielens 100k
[Gorla et al, 2013]
Movielens 1M
[Yin et al, 2012]
Movielens 100k, SVD
We need to understand why this happens

Goal of this tutorial
•  Identify the steps that can act as hurdles when
replicating experimental results
– Focusing on the speciﬁc details inherent to the
recommender systems
•  We will analyze this problem using the
following representation of a generic
recommender system process

Train
Test
Validation
Splitting
Recommender
generates
ranking
(for a user)
prediction
for a given
item (& user)
Ranking
Prediction
Coverage
Diversity
.
.
.
Evaluator
Collector
Stats

In this tutorial
•  We will focus on replication and
reproducibility
– Deﬁne the context
– Present typical setting and problems
– Propose some guidelines
– Exhibit the most typical scenarios where
experimental results in recommendation may
hinder replication

NOT in this tutorial
•  Deﬁnition of evaluation in recommendation:
– In-depth analysis of evaluation metrics
– Novel evaluation dimensions
– User evaluation
Ø Wednesday’s lectures on evaluation

Outline
•  Demo

Reproducible Experimental Design
•  We need to distinguish
–  Replicability
–  Reproducibility
•  Different aspects:
–  Algorithmic
–  Published results
–  Experimental design
•  Goal:
–  to have an environment for reproducible experiments

Deﬁnition:
Replicability
To copy something
•  The results
•  The data
•  The approach
Being able to evaluate in
the same setting and obtain
the same results

Deﬁnition:
Reproducibility
To recreate something
•  The (complete) set of
experiments
•  The (complete) set of
results
•  The (complete)
experimental setup
To (re)launch it in production
with the same results

Comparing against the state-of-the-art
Your settings are not exactly
like those in paper X, but it is
a relevant paper
Congrats, you’re
done!
Congrats!You have shown that
paper X behaves different in
the new setting
There is something wrong/
incomplete in the experimental
design.Try again!
They agree
They do not agree
Do results
match the
original
paper?
Yes!
No!
Do results
agree with
original
paper?

Comparing against the state-of-the-art
Your settings are not exactly
like those in paper X, but it is
a relevant paper
Reproduce
results of paper X
Congrats, you’re
done!
Replicate results
of paper X
Congrats!You have shown that
paper X behaves different in
the new setting
There is something wrong/
incomplete in the experimental
design.Try again!
They agree
They do not agree
Do results
match the
original
paper?
Yes!
No!
Do results
agree with
original
paper?

What about Reviewer 3?
•  “It would be interesting to see this done on a
different dataset…”
–  Repeatability
–  The same person doing the whole pipeline over again
•  “How does your approach compare to [Reviewer 3 et
al. 2003]?”
–  Reproducibility or replicability (depending on how similar
the two papers are)

Repeat vs. replicate vs. reproduce vs. reuse

Motivation for reproducibility
In order to ensure that our experiments, settings,
and results are:
–  Valid
–  Generalizable
–  Comparable
–  Of use for others
–  etc.
we must make sure that others can reproduce our
experiments in their setting

Making reproducibility
easier
•  Description, description,
description
•  No magic numbers
•  Specify values for all parameters
•  Motivate!
•  Keep a detailed protocol of
everything
•  Describe process clearly
•  Use standards
•  Publish code (nobody expects you
to be an awesome developer,
you’re a researcher)
•  Publish data
•  Publish supplemental material

Replicability, reproducibility, and progress
•  Can there be actual progress if no valid comparison
can be done?
•  What is the point of comparing two approaches if
the comparison is ﬂawed?
•  How do replicability and reproducibility facilitate
actual progress in the ﬁeld?

Summary
•  Important issues when running experiments
–  Validity of results (replicability)
–  Comparability of results (reproducibility)
–  Validity of experimental setup (repeatability)
•  We need to incorporate reproducibility and
replication to facilitate progress in the ﬁeld
•  If your research is reproducible for others, it has
more value

Outline
–  Dataset collection
–  Splitting
–  Recommender algorithms
–  Candidate items
–  Evaluation metrics
–  Statistical testing
•  Demo

Replication in Recommender Systems
•  Replicability/reproducibility/repeatability: useful and
desirable in any ﬁeld
–  How can they be addressed when dealing with
recommender systems?
•  Proposal: analyze the recommendation process and
identify each stage that may affect the ﬁnal results

Train
Test
Validation
Splitting
Recommender
generates
ranking
(for a user)
prediction
for a given
item (& user)
Ranking
Prediction
Coverage
Diversity
.
.
.
Evaluator
Collector
Stats

DATA CREATION AND
COLLECTION

What is a dataset?

Public datasets
•  Movielens 20M
–  “Users were selected at random for inclusion.All selected users had rated at least 20
movies.”
•  Netﬂix Prize
–  Details withheld
•  Xing (RecSys Challenge 2016/2017)
–  Details withheld
•  Last.fm (360k, MSD)
–  Undocumented cleaning applied
•  MovieTweetings
–  All IMDb ratings… from Twitter
–  2nd hand information

Creating your own datasets
•  Ask yourself:
–  What are we collecting?
–  How are we collecting it?
•  How should we be collecting it?
–  Are we collecting all (vital) interactions?
•  dwell time vs. clicks vs. comments vs. swipes vs.
likes vs. etc.
–  Are we documenting the process in sufﬁcient detail?
–  Are we sharing the dataset in a format understood by
others (and supported by software)?

The user-item matrix
User Item Interac,on Timestamp
1 1 1 2017-…
1 2 1 …
2 3 2 …

Releasing the dataset
•  Make the dataset publicly available
– Otherwise your work is not reproducible
•  Provide an in-depth overview
– Website, paper, etc.
•  Communicate it
– Mailing lists, RecSysWiki, website, etc.

Releasing the dataset
•  Consider releasing ofﬁcial training, test,
validation splits.
•  Present baseline algorithm results for released
splits.
•  Have code examples of how to work with the
data (splits, evaluations, etc.)

Train
Test
Validation
Splitting
Recommender
generates
ranking
(for a user)
prediction
for a given
item (& user)
Ranking
Prediction
Coverage
Diversity
.
.
.
Evaluator
Collector
Stats

DATA SPLITTING AND
PREPARATION

Train
Test
Validation
Splitting

Train
Test
Validation
Splitting•  Sizes?
•  How to split?
•  Filtering?
•  How to document?

Train
Test
Validation
Splitting•  Sizes?
•  How to split?
•  Filtering?
•  How to document?
•  What’s the task?
–  Rating prediction
–  Top-n
•  What’s important for the
algorithm?
–  Time
–  Relevance

Train
Test
Validation
Splitting•  Which are the candidate items that we will be
recommending?
•  Who are the candidate users we will be
recommending (and evaluating) for?
•  Do we have any limitations on numbers?
–  Cold start?
–  Temporal/trending recommendations?
–  Other?

Scenarios
•  Random
•  All users at once
•  All items at once
•  One user at once
•  One item at once
•  Temporal
•  Temporal for one user
•  Relevance thresholds

Random
•  The split does not take
into consideration
–  Whether users or items
are left out of the
training or test sets.
–  The relevance of items
–  The scenario of the
recommendation

All users at once
into consideration
–  Whether items are left
out of the training or
test sets.
•  Can take into
consideration
–  The relevance of items
(per user or in general)

All items at once
into consideration
–  Whether users are left
out of the training or
test sets.

One user at once
•  The split takes into
consideration
–  The interactions of all
other users when
creating the splits for
one speciﬁc user
•  Resulting training set
contains all other
user-item interactions

One item at once
consideration
–  The interactions of all
other users when
creating the splits for
one speciﬁc item
•  Resulting training set
contains all other
user-item interactions

Temporal
consideration
–  The timestamp of
interactions
•  All items newer than a
certain timestamp are
discarded part of the
test set.

Filters
•  What ﬁlters?
•  Why ﬁlters?
•  Removing items/users with few interactions
creates a skewed dataset
– Sometimes this is a good thing
– Needs proper motivation

Implementation
•  Most recommender system frameworks
implement some form of splitting
however
•  Documenting what choices were selected for the
splitting is crucial for the work to be
reproducible. Even when using established
frameworks

Data splitting - LensKit
h)p://lenskit.org/documenta=on/evaluator/data/

Data splitting - LibRec
h)ps://www.librec.net/dokuwiki/doku.php?id=DataModel
5525-Aug-17

5625-Aug-17

5725-Aug-17

5825-Aug-17

5925-Aug-17

6025-Aug-17

Partitions
Dataset
Test set
Training set Test set Valida=on set

Partitions
Dataset
Test set
Training set Test set Valida=on set
Training set

Train
Test
Validation
Splitting
Recommender
generates
ranking
(for a user)
prediction
for a given
item (& user)
Ranking
Prediction
Coverage
Diversity
.
.
.
Evaluator
Collector
Stats

RECOMMENDATION

The recommender

Deﬁning the recommender
•  Many versions of the same thing
•  Various implementations/design choices for
– Collaborative Filtering
– Similarity/distance measures
– Factorization techniques (MF/FM)
– Probabilistic modeling

Design
•  There are multiple ways of implementing the same
algorithms, similarities, metrics.
•  Irregularities arise even when using a known
implementation (from an existing framework)
–  Look at and understand the source code
–  Report implementational variations (esp. when comparing
to others)
•  Magic numbers
•  Rounding errors
•  Thresholds
•  Optimizations

Collaborative Filtering

Collaborative Filtering
•  Both equations are usually referred to using
the same name, i.e. k-nearest neighbor, user-
based, cf.

Similarities
•  Similarity metrics may have different design
choices as well
– Normalized (by user) ratings/values
– Shrinking parameter

CF Common Exceptions
•  Two users having one rating/interaction each
(same item)
•  Both have liked it/rated similarly
•  What is the similarity of these two?

CF Common Exceptions
•  Two users having rated 500 items each
•  5 item intersect and have the same ratings/
values
•  What is the similarity of these two?

CF Implementation
•  LensKit
– No matter the recommender chosen, there is
always a backup recommender to your chosen
one. (BaselineScorer)
– If your chosen recommender cannot ﬁll your list
of recommender items, the backup recommender
will do so instead.

CF Implementation
•  RankSys
– Allows setting a similarity exponent, making the
similarity stronger/weaker than normal
– Similarity score defaults to 0.0

CF Implementation
•  LibRec
– Defaults to global mean when it cannot predict a
rating for a user

Matrix Factorization
•  Ranking vs. Rating prediction
– Implementations vary between various
frameworks.
– Some frameworks contain several
implementations of the same algorithms to tender
to ranking speciﬁcally or rating prediction
speciﬁcally

MF Implementation
•  RankSys
–  Bundles probabilistic modeling (PLSA) with matrix
factorization (ALS)
–  Has three parallel ALS implementations
•  Generic ALS
•  Y. Hu,Y. Koren, C.Volinsky. Collaborative ﬁltering for implicit
feedback datasets. ICDM 2008
•  I. Pilászy, D. Zibriczky and D.Tikk. Fast ALS-based Matrix
Factorization for Explicit and Implicit Feedback Datasets. RecSys
2010.

MF Implementation
•  LibRec
– Separate rating prediction and ranking models
– RankALSRecommender - Ranking
•  Takács andTikk. Alternating Least Squares for Personalized
Ranking. RecSys 2012.
– MFALSRecommender – Rating Prediction
•  Zhou et al. Large-Scale Parallel Collaborative Filtering for the
Netﬂix Prize.AAIM 2008

Probabilistic Modeling
•  Various ways of implementing the same
algorithm
– LDA using Gibbs sampling
– LDA using variational Bayes

Probabilistic Implementation
•  RankSys
– Uses Mallet’s LDA implementation
– Newman et al. 2009. Distributed Algorithms forTopic
Models.

Probabilistic Implementation
•  LibRec
– LDA for implicit feedback
– Grifﬁths. 2002. Gibbs sampling in the generative
model of Latent Dirichlet Allocation

Recommending: LibRec
h)ps://www.librec.net/dokuwiki/doku.php?id=Recommender
84

Recommending: LibRec
h)ps://www.librec.net/dokuwiki/doku.php?id=Recommender
h)ps://github.com/guoguibing/librec/issues/76
85

Recommending LensKit
8625-Aug-17

Outline
–  Splitting
–  Recommender algorithm
•  Demo

Train
Test
Validation
Splitting
Recommender
generates
ranking
(for a user)
prediction
for a given
item (& user)
Ranking
Prediction
Coverage
Diversity
.
.
.
Evaluator
Collector
Stats

Candidate item generation
Different ways to select candidate items to be
ranked:
TestRatings: rated items by u in test set
TestItems: every item in test set
TrainingItems: every item in training
AllItems: all items in the system
Note: in CF,AllItems and TrainingItems
produce the same results
Solid triangle represents the target user.
Boxed ratings denote test set.

Different ways to select candidate items to be
ranked:
0
0.05
0.30
0.35
0.40
TR 3 TR 4 TeI TrI AI OPR
P@50 SVD50
IB
UB50
0
0.20
0.90
0.10
0.30
1.00
TR 3 TR 4 TeI TrI AI OPR
Recall@50
0
0.90
0.10
0.05
0.80
1.00
TR 3
Solid triangle represents the target user.
Boxed ratings denote test set.
[Bellogín et al, 2011]

[Said & Bellogín, 2014]
Impact of different strategies for candidate item
selection:
RPN: RelPlusN
a ranking with
1 relevant and
N non-relevant items
UT: UserTest
same as TestRatings

Impact of test size for different candidate item
selection strategies:
The actual value of the metric may be affected by the amount of
known information
RPN RPNTestItems

Outline
•  Focus on Recommender Systems
–  Splitting
•  Demo

Train
Test
Validation
Splitting
Recommender
generates
ranking
(for a user)
prediction
for a given
item (& user)
Ranking
Prediction
Coverage
Diversity
.
.
.
Evaluator
Collector
Stats

considering
Evaluation metric computation
When coverage is not complete, how are the
metrics computed?
–  If a user receives 0 recommendations
Option a:
Option b:
MAE = Mean Absolute Error
RMSE = Root Mean Squared Error

metrics computed?
Option a:
Option b:
User
rec1 rec2
#recs metric(u) #recs metric(u)
u1 5 0.8 5 0.7
u2 3 0.2 5 0.5
u3 0 -- 5 0.3
u4 1 1.0 5 0.7
Option a 0.50 0.55
Option b 0.66 0.55
User
coverage
3/4 4/4

metrics computed?
–  If a value is not predicted (esp. for error-based
metrics)

metrics computed?
–  If a value is not predicted (esp. for error-based
metrics)
User-item
pairs
Real Rec1 Rec2 Rec3
(u1, i1) 5 4 NaN 4
(u1, i2) 3 2 4 NaN
(u1, i3) 1 1 NaN 1
(u2, i1) 3 2 4 NaN
MAE/RMSE, ignoring NaNs 0.75/0.87 2.00/2.00 0.50/0.70
MAE/RMSE, NaNs as 0 0.75/0.87 2.00/2.65 1.75/2.18
MAE/RMSE, NaNs as 3 0.75/0.87 1.50/1.58 0.25/0.50

Variations on metrics:
Error-based metrics can be normalized or averaged per user:
–  Normalize RMSE or MAE by the range of the ratings
(divide by rmax – rmin)
–  Average RMSE or MAE to compensate for unbalanced
distributions of items or users

nDCG has at least two discounting functions
(linear and exponential decay)

Ranking-based metrics are usually computed up to
a ranking position or cutoff k
P = Precision (Precision at k)
R = Recall (Recall at k)
MAP = Mean Average Precision
Is the cutoff being reported? Are the metrics computed until the
end of the list? Is that number the same across all the users?

If ties are present in the ranking scores, results
may depend on the implementation

Internal evaluation methods of different frameworks (Mahout
(AM), LensKit (LK), MyMediaLite (MML)) present different
implementations of these aspects
25-Aug-17 ACM RecSys Summer School 2017 107[Said & Bellogín, 2014]

Decisions (implementations) found in some
recommendation frameworks:
Regarding coverage:
•  LK and MML use backup recommenders
•  LR: not actual backup recommender, but default values (global
mean) are provided when not enough neigbors (or all similarities
are negative) are found in KNN
•  RS allows to average metrics explicitly by option a or b (see
different constructors of AverageRecommendationMetric)
LK: https://github.com/lenskit/lenskit
LR: https://github.com/guoguibing/librec
MML: https://github.com/zenogantner/MyMediaLite
RS: https://github.com/RankSys/RankSys

Regarding metric variations:
•  LK, LR, MML use a logarithmic discount for nDCG
•  RS also uses a logarithmic discount for nDCG, but relevance is
normalized with respect to a threshold
•  LK does not take into account predictions without scores for
error metrics
•  LR fails if coverage is not complete for error metrics
•  RS does not compute error metrics
LK: https://github.com/lenskit/lenskit
LR: https://github.com/guoguibing/librec
MML: https://github.com/zenogantner/MyMediaLite
RS: https://github.com/RankSys/RankSys

Regarding candidate item generation:
•  LK allows defining a candidate and exclude set
•  LR: delegated to the Recommender class.
AbstractRecommender defaults to TrainingItems
•  MML allows different strategies: training, test, their overlap and
union, or a explicitly provided candidate set
•  RS defines different ways to call the recommender: without
restriction, with a list size limit, with a filter, or with a candidate
set

Regarding ties:
•  MML: not deterministic (Recommender.Recommend sorts
items by descending score)
•  LK: depends on using predict package (not deterministic:
LongUtils. keyValueComparator only compares scores) or
recommend (same ranking as returned by algorithm)
•  LR: not deterministic (Lists.sortItemEntryListTopK only
compares the scores)
•  RS: deterministic (IntDoubleTopN compares values and then
keys)

Outline
•  Focus on Recommender Systems
–  Splitting
•  Demo

Train
Test
Validation
Splitting
Recommender
generates
ranking
(for a user)
prediction
for a given
item (& user)
Ranking
Prediction
Coverage
Diversity
.
.
.
Evaluator
Collector
Stats

Statistical testing
Make sure the statistical testing method is
reported
– Paired/unpaired test, effect size, conﬁdence
interval
– Specify why this speciﬁc method is used
– Related statistics (such as mean, variance,
population size) are useful to interpret the results
[Sakai, 2014]

Statistical testing
When doing cross-validation, there are several
options to take the samples for the test:
–  One point for each aggregated value of the metric
•  Very few points (one per fold)
–  One point for each value of the metric, on a user basis
•  If we compute a test for each fold, we may ﬁnd
inconsistencies
•  If we compute a test with all the (concatenated) values, we
may distort the test: many more points, not completely
independent
[Bouckaert, 2003]

Replication and Reproducibility in RecSys:
Summary
•  Reproducible experimental results depend on acknowledging
every step in the recommendation process
–  As black boxes so every setting is reported
–  Applies to data collection, data splitting, recommendation, candidate
item generation, metric computation, and statistics
•  There exist several details (in implementation) that might hide
important effects in ﬁnal results

Key takeaways
•  Every decision has an impact
–  We should log every step taken in the experimental
part and report that log
•  There are more things besides papers
–  Source code, web appendix, etc. are very useful to
provide additional details not present in the paper
•  You should not fool yourself
–  You have to be critical about what you measure and
not trust intermediate “black boxes”

Next steps?
•  We should agree on standard implementations, parameters,
instantiations, …
–  Example: trec_eval in IR

Next steps?
instantiations, …
•  Replicable badges for journals / conferences

Next steps?
instantiations, …
h)p://
valida=on.scienceexchange.com
The aim of the Reproducibility Ini=a=ve is to iden=fy and reward high quality
reproducible research via independent valida=on of key experimental results

Next steps?
instantiations, …
•  Investigate how to improve reproducibility

Next steps?
instantiations, …
•  Investigate how to improve reproducibility
•  Benchmark, report, and store results

Pointers
•  Email and Twitter
–  Alejandro Bellogín
•  alejandro.bellogin@uam.es
•  @abellogin
–  Alan Said
•  alansaid@acm.org
•  @alansaid
•  Slides:
•  https://github.com/recommenders/rsss2017
125

RiVal
Recommender System Evaluation
Toolkit
http://rival.recommenders.net
http://github.com/recommenders/rival
126

References and Additional reading
•  [Armstrong et al, 2009] Improvements That Don’t Add Up:Ad-Hoc Retrieval Results Since 1998.
CIKM
•  [Bellogín et al, 2010] A Study of Heterogeneity in Recommendations for a Social Music Service.
HetRec
•  [Bellogín et al, 2011] Precision-Oriented Evaluation of Recommender Systems: an Algorithm
Comparison. RecSys
•  [Bellogín et al, 2013] An Empirical Comparison of Social, Collaborative Filtering, and Hybrid
Recommenders.ACM TIST
•  [Bellogín et al, 2017] Statistical biases in Information Retrieval metrics for recommender systems.
Information Retrieval Journal
•  [Ben-Shimon et al, 2015] RecSys Challenge 2015 and theYOOCHOOSE Dataset. RecSys
•  [Bouckaert, 2003] Choosing Between Two Learning Algorithms Based on Calibrated Tests. ICML
•  [Cremonesi et al, 2010] Performance of Recommender Algorithms on Top-N Recommendation
Tasks. RecSys
•  [Filippone & Sanguinetti, 2010] Information Theoretic Novelty Detection. Pattern Recognition
•  [Fleder & Hosanagar, 2009] Blockbuster Culture’s Next Rise or Fall:The Impact of Recommender
Systems on Sales Diversity. Management Science
128

•  [Ge et al, 2010] Beyond accuracy: evaluating recommender systems by coverage and serendipity.
RecSys
•  [Gorla et al, 2013] Probabilistic Group Recommendation via Information Matching.WWW
•  [Herlocker et al, 2004] Evaluating Collaborative Filtering Recommender Systems.ACM Transactions
on Information Systems
•  [Jambor & Wang, 2010] Goal-Driven Collaborative Filtering. ECIR
•  [Knijnenburg et al, 2011] A Pragmatic Procedure to Support the User-Centric Evaluation of
Recommender Systems. RecSys
•  [Koren, 2008] Factorization Meets the Neighborhood: a Multifaceted Collaborative Filtering Model.
KDD
•  [Lathia et al, 2010] Temporal Diversity in Recommender Systems. SIGIR
•  [Li et al, 2010] Improving One-Class Collaborative Filtering by Incorporating Rich User Information.
CIKM
•  [Pu et al, 2011] A User-Centric Evaluation Framework for Recommender Systems. RecSys
•  [Said & Bellogín, 2014] Comparative Recommender System Evaluation: Benchmarking
Recommendation Frameworks. RecSys
•  [Sakai, 2014] Statistical reform in Information Retrieval? SIGIR Forum
129

•  [Schein et al, 2002] Methods and Metrics for Cold-Start Recommendations. SIGIR
•  [Shani & Gunawardana, 2011] Evaluating Recommendation Systems. Recommender Systems
Handbook
•  [Steck & Xin, 2010] A Generalized Probabilistic Framework and itsVariants for Training Top-k
Recommender Systems. PRSAT
•  [Tikk et al, 2014] Comparative Evaluation of Recommender Systems for Digital Media. IBC
•  [Vargas & Castells, 2011] Rank and Relevance in Novelty and Diversity Metrics for Recommender
Systems. RecSys
•  [Weng et al, 2007] Improving Recommendation Novelty Based on Topic Taxonomy.WI-IAT
•  [Yin et al, 2012] Challenging the Long Tail Recommendation.VLDB
•  [Zhang & Hurley, 2008] Avoiding Monotony: Improving the Diversity of Recommendation Lists.
RecSys
•  [Zhang & Hurley, 2009] Statistical Modeling of Diversity in Top-N Recommender Systems.WI-IAT
•  [Zhou et al, 2010] Solving the Apparent Diversity-Accuracy Dilemma of Recommender Systems.
PNAS
•  [Ziegler et al, 2005] Improving Recommendation Lists Through Topic Diversiﬁcation.WWW
130

Replication of Recommender Systems Research

More Related Content

What's hot (20)

Similar to Replication of Recommender Systems Research (20)

More from Alan Said (13)

Recently uploaded (20)

Replication of Recommender Systems Research