An Iterative Distance-Based Model for Unsupervised Weighted Rank Aggregation

An Iterative Distance-Based Model for
Unsupervised Weighted Rank Aggregation
Leonidas Akritidis1, Athanasios Fevgas1, Panayiotis Bozanis2,*
1Department of Electrical and Computer Engineering, University of Thessaly
2Department of Science and Technology, International Hellenic University
*Research conducted while being in 1
The 2019 IEEE/WIC/ACM International Conference on
Web Intelligence (WI 2019)
October 14-17, 2019, Thessaloniki, Greece

Rank Aggregation (RA)
• A set of n voters V.
• Given a query or a subject, each voter v
submits a single ranked list Rv of
answers/suggestions .
• A rank aggregation method T fuses all Rv lists
into one aggregated list L with improved
ranking of its elements.
• Applications: Voting systems, bioinformatics,
Web metasearch, collaborative filtering, etc.
L. Akritidis, A. Fevgas, P. Bozanis 2IEEE/WIC/ACM WI 2019, October 14-17, 2019, Thessaloniki, Greece
v
ir

Weighted vs. non weighted RA
• Each voter v is assigned a weight wv.
• Non weighted rank aggregation:
– All voters are treated equally.
–
• Weighted rank aggregation: Each voter is
assigned a weight which reflects his/her
importance and/or expertise on the subject.
–
1,vw v V  

Motivation
• Who is the best football player?
• What does this tell us about v1?
– All of his/her suggestions made it to the top-3 of L.
– S/he is an expert on the subject.
• What about v3?
– Only one of his/her suggestions in the top-3 of L.
– S/he is not so familiar with the subject.
Voter 1 v1
Messi
Ronaldo
Neymar
Voter 2 v2
Ronaldo
Messi
Sallah
Voter 3 v3
Messi
Sallah
Van Dijk
Voter 4 v4
Neymar
Mane
Messi
Player Borda Score
Messi 3+2+3+2=10
Ronaldo 2+3+0+0=5
Neymar 1+0+0+3=4
Sallah 0+1+2+0=3
Mane 0+0+0+2=2
VanDijk 0+0+1+0=1

Idea
• Compare (find the distance of) each input list Rv
with the aggregate list L.
• The voter who submits a ranked list that has a
small distance from L is considered as an expert.
– His/her choices have strong support by other voters.
– S/he should be assigned a high weight.
• In contrast: The voter who submits a ranked list
that has a great distance from L is not an expert.
– His/her weight should be smaller compared to the
previous voter.

Ranked List Distance Metrics
• Spearman Footrule distance:
• Kendall’s tau:
• Scaled Footrule Distance:
• Proposed: Locality-Sensitive Scaled Footrule distance takes
into account not only the (scaled) difference in rankings,
but also their location in the aggregate list. It “punishes”
the discrepancies which occur in high positions of L.

Distance-Based Model for URA
• Assign all voters an initial equal weight wv,0.
• Apply the RA method T of choice in the traditional,
non weighted fashion.
• Obtain an initial aggregate list L0.
• For each voter v, compute the distance d(Rv, L0)
between his/her list Rv and L0.
• Compute the new weight wv,1 of v according to:
• f is a kernel function that depends on d(Rv, L0).
• Reapply T on its weighted form.

Iterative Distance-Based Model
• The previous equation can be applied many times
in an iterative fashion until the weights converge.
• The kernel function f must:
– be asymptotically upper bounded, and
– Satisfy if
• Exp Kernel function:
• i: the ith iteration
• Finally:
   , ,v v
d R L d R L
     , ,v v
f d R L f d R L

     , exp ,v v
f d R L i d R L 

Initialize the weights of the voters
Apply non-weighted RA T
Iteration Counter
Flag: Checks convergence of all weights
Apply weighted RA T
Compute the distance
between the input list Rv of
a voter v and the aggregate
list L. Update the weight wv
of v and check convergence.
Iterateuntilconvergence

Experiments
• We utilized 6 datasets from the Web Tracks of TREC
2009-2014.
• In these tracks, the participant groups are given a
set of 50 predefined queries.
• They submit their result lists which are evaluated
by using relevance judgments from human judges.
• Ideal for rank aggregation.
• Fair, unbiased, reputable.

Results (Mean Average Precision)
• BC: Borda Count, CM: Condorcet Method, OA: Outranking
Approach (Vanderpooten et.al, ACM SIGIR 2011).
• UWI: Unsupervised, Weighted, Iterative.
– SF: Locality-Sensitive Scaled Footrule Distance.
– F: Standard Scaled Footrule Distance.
• Our model achieves a consistent improvement of about 4-10%.

Results (other evaluation metrics)

Preliminary Convergence Study
• The proposed distance metric needs on average 1-2
more iterations compared to the original scaled footrule
distance.
• The weights converge more slowly for Borda Count.
Borda Count is much faster than Condorcet Method and
Outranking Approach.

Conclusions
• We presented a new unsupervised model for
weighted rank aggregation.
• The model considers that the voters whom lists are
close to the aggregate list as experts.
• It iteratively computes the distance of each input list
with the aggregate list and modifies the voter
weights accordingly.
• We introduced a metric for computing the distance
between two ranked lists.
– Sensitive to the “locality” of disagreements.
• The model was evaluated by using datasets from
TREC and was found to improve rankings by 4-10%.

Thank you
Any questions?

An Iterative Distance-Based Model for Unsupervised Weighted Rank Aggregation

More Related Content

Recently uploaded (20)

Featured (20)

An Iterative Distance-Based Model for Unsupervised Weighted Rank Aggregation