Online Testing Learning to Rank with Solr Interleaving

Online Testing Learning to Rank
with Solr Interleaving
 
Alessandro Benedetti, Director
24th
March 2022

‣ Born in Tarquinia(ancient Etruscan city in Italy)
‣ R&D Software Engineer
‣ Director
‣ Master in Computer Science
‣ PC member for ECIR, SIGIR and Desires
‣ Apache Lucene/Solr PMC member/committer
‣ Elasticsearch expert
‣ Semantic, NLP, Machine Learning
technologies passionate
‣ Beach Volleyball player and Snowboarder
Who We Are
Alessandro Benedetti

‣ Headquarter in London/distributed
‣ Open Source Enthusiasts
‣ Apache Lucene/Solr experts
‣ Elasticsearch experts
‣ Community Contributors
‣ Active Researchers
‣ Hot Trends : Neural Search,
Natural Language Processing
Learning To Rank,
Document Similarity,
Search Quality Evaluation,
Relevancy Tuning
SEArch SErvices
www.sease.io

Overview
Learning to Rank
Online Testing for Business
Interleaving
Apache Solr Implementation
DEMO

Learning from user implicit/explicit feedback
To
Rank documents (sensu lato)
These types of models focus more on the relative ordering of items rather
than the individual label (classiﬁcation) or score (regression), and are
categorized as Learning To Rank models.
What is it?

• [sci-fi] Sentient system that learn by itself
“Machine Learning stands for that, doesn’t it?” Unknown
• [Integration] Easy to set up and tune it -> it takes patience, time and multiple
experiments
• [Explainability] Easy to give a human understandable explanation of why the model
operates in certain ways
What is not

• Ranking is a central part of many information retrieval problems, such as document
retrieval, collaborative ﬁltering, sentiment analysis, and online advertising.
Application

“Learning to rank is the application of machine
learning, typically supervised, semi-supervised or
reinforcement learning, in the construction of
ranking models for information retrieval
systems.” Wikipedia
Learning To Rank Users Interactions
Logger
Judgement Collector
UI
Interactions
Training
Training Data

An incorrect or imperfect test set brings us model evaluation results that
aren’t reflecting the real model improvement/regressions
e.g.
We may get an extremely high evaluation metric offline, but only because
we improperly designed the test, even if the model is unfortunately not a
good fit.
There are several problems that are hard to be detected
with an offline evaluation:
[Online] A Business Perspective

An incorrect test set allow us to obtain model evaluation
results that aren’t reflecting the real model improvement.
One sample per query group
One relevance label for all the samples of a query
group
Interactions considered for the data set creation

A incorrect test set allow us to obtain model evaluation
Finding a direct correlation between the offline
evaluation metrics and the parameters used for the online
model performance evaluation (e.g. revenues, click
through rate…).

A incorrect test set allow us to obtain model evaluation
Finding a direct correlation between the offline
evaluation metrics and the parameters used for the online
model performance evaluation (e.g. revenues).
Is based on generated relevance labels that not always
reflect the real user need.

The reliability of the results: we directly observe the user
behaviour.
The interpretability of the results: we directly observe the
impact of the model in terms of online metrics the business
cares.
The possibility to observe the model behavior: we can see
how the user interact with the model and figure out how to
improve it.
Using online testing can lead to many advantages:
[Online] Business Advantages

50%
50%
A B
20% 40%
Control Variation
[Online] A/B testing

100%
Model A Model B
2
1 3 1 2 3
1 2 3 4
Interleaving

[Online] Interleaving results estimator
>0 winner A
<0 winner B
=0 tie

● It reduces the problem with users’ variance due to their
separation in groups (group A and group B).
● It is more sensitive when comparing models.
● It requires less traffic.
● It requires less time to achieve reliable results.
● It doesn’t necessarily expose a bad model to a sub
population of users.
Interleaving Advantages

There are different types of interleaving:
Balanced Interleaving: alternate insertion with one model
having the priority(decided at the beginning of the
interleaving().
Balanced Interleaving

having the priority.
DRAWBACK
When comparing two very similar models.
Model A: lA
= (a, b, c, d)
Model B: lB
= (b, c, d, a)
The comparison phase will bring the Model B to win more
often than Model A. This happens regardless of the model
chosen as prior.
This drawback arises due to:
the way in which the evaluation of the results is done.
the fact that model_B rank higher than model_A all documents
with the exception of a.
Balanced Interleaving

Team-Draft Interleaving: method of team captains in
team-matches.
Team-Draft Interleaving

Team-Draft Interleaving: method of team capitains in
team-matches.
DRAWBACK
When comparing two very similar models.
Model A: lA
= (a, b, c, d)
Model B: lB
= (b, c, d, a)
Suppose c to be the only relevant document.
With this approach we can obtain four different interleaved lists:
lI1
= (aA
, bB
, cA
, dB
)
lI2
= (bB
, aA
, cB
, dA
)
lI3
= (bB
, aA
, cA
, dB
)
lI4
= (aA
, bB
, cB
, dA
)
All of them putting c at the same rank.
Tie!
But Model B should be chosen
as the best model!
[Online] Team-Draft Interleaving

team-matches.
Probabilistic Interleaving: rely on probability distributions.
Every documents have a non-zero probability to be added in
the interleaved result list.
[Online] Probabilistic Interleaving

team-matches.
Probabilistic Interleaving: rely on probability distributions.
Every documents have a non-zero probability to be added in
the interleaved result list.
DRAWBACK
The use of probability distribution could lead to a worse user
experience. Less relevant document could be put higher.
[Online] Probabilistic Interleaving
https://sease.io/2020/05/online-testing-for-learning-to-rank-interleaving.html

• Include the required contrib JARs. Note that by default paths are relative to the Solr core so they may need
adjustments to your configuration, or an explicit specification of the $solr.install.dir.
<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-ltr-d.*.jar" />
• Declaration of the ltr query parser.
<queryParser name="ltr" class="org.apache.solr.ltr.search.LTRQParserPlugin"/>
• Configuration of the feature values cache.
<cache name="QUERY_DOC_FV"
class="solr.search.LRUCache"
size="4096"
initialSize="2048"
autowarmCount="4096"
regenerator="solr.search.NoOpRegenerator" />
Minimum Requirements

• Declaration of the [features] transformer.
<transformer name="features"
class="org.apache.solr.ltr.response.transform.LTRFeatureLoggerTransformerFactory">
<str name="fvCacheName">QUERY_DOC_FV</str>
</transformer>
• Declaration of the [interleaving] transformer.
<transformer name="interleaving"
class="org.apache.solr.ltr.response.transform.LTRInterleavingTransformerFactory"/>
Minimum Requirements

Implements the ranking function:
General form Class Specific examples
Linear LinearModel RankSVM, Pranking
Multiple Additive Trees MultipleAdditiveTreesModel LambdaMART, Gradient
Boosted Regression Trees
(GBRT)
Neural Network NeuralNetworkModel RankNet
(wrapper) DefaultWrapperModel (not applicable)
(custom) (custom class extending
AdapterModel)
(not applicable)
(custom) (custom class extending
LTRScoringModel)
(not applicable)
Models

‣ Computes the scores using a dot product
https://lucene.apache.org/solr/8_8_0//solr-ltr/org/apache/solr/ltr/model/LinearModel.html
Example configuration:
{
"class" : "org.apache.solr.ltr.model.LinearModel",
"name" : "myModelName",
"features" : [
{ "name" : "userTextTitleMatch" },
{ "name" : "originalScore" },
{ "name" : "isBook" }
],
"params" : {
"weights" : {
"userTextTitleMatch" : 1.0,
"originalScore" : 0.5,
"isBook" : 0.1
}
}
}
Linear

‣ computes scores based on the summation of multiple weighted trees
https://lucene.apache.org/solr/8_8_0//solr-ltr/org/apache/solr/ltr/model/MultipleAdditiveTreesModel.html
{
"class" : "org.apache.solr.ltr.model.MultipleAdditiveTreesModel",
"name" : "multipleadditivetreesmodel",
"features":[
{ "name" : "userTextTitleMatch"},
{ "name" : "originalScore"}
],
}
"params" : {
"trees" : [
{
"weight" : "1",
"root": {
"feature" : "userTextTitleMatch",
"threshold" : "0.5",
"left" : {
"value" : "-100"
},
"right" : {
"feature" : "originalScore",
"threshold" : "10.0",
"left" : {
"value" : "50"
},
"right" : {
"value" : "75"
}
}
}
},
{
"weight" : "2",
"root" : {
"value" : "-10"
}
}
]
}
Multiple Additive Trees Model

‣ computes scores using a neural network.
https://lucene.apache.org/solr/8_8_0//solr-ltr/org/apache/solr/ltr/model/MultipleAdditiveTreesModel.html
{
"class" : "org.apache.solr.ltr.model.NeuralNetworkModel",
"name" : "rankNetModel",
"features" : [
{ "name" : "documentRecency" },
{ "name" : "isBook" },
{ "name" : "originalScore" }
],
"params" : {
"layers" : [
{
"matrix" : [ [ 1.0, 2.0, 3.0 ],
[ 4.0, 5.0, 6.0 ],
[ 7.0, 8.0, 9.0 ],
[ 10.0, 11.0, 12.0 ] ],
"bias" : [ 13.0, 14.0, 15.0, 16.0 ],
"activation" : "sigmoid"
},
{
"matrix" : [ [ 17.0, 18.0, 19.0, 20.0 ],
[ 21.0, 22.0, 23.0, 24.0 ] ],
"bias" : [ 25.0, 26.0 ],
"activation" : "relu"
},
{
"matrix" : [ [ 27.0, 28.0 ],
[ 29.0, 30.0 ] ],
"bias" : [ 31.0, 32.0 ],
"activation" : "leakyrelu"
},
{
"matrix" : [ [ 33.0, 34.0 ],
[ 35.0, 36.0 ] ],
"bias" : [ 37.0, 38.0 ],
"activation" : "tanh"
},
{
"matrix" : [ [ 39.0, 40.0 ] ],
"bias" : [ 41.0 ],
"activation" : "identity"
}
]
}
Neural Network

http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr
model=myModel reRankDocs=100}&fl=id,score
To obtain the feature values computed during reranking, add [features] to the ﬂ parameter, for
example:
model=myModel reRankDocs=100}&fl=id,score,[features]
To rerank using external feature information:
model=myEfiModel efi.text=test efi.preferredManufacturer=Apache
efi.fromMobile=0 efi.answer=13}&fl=id,cat,manu,score,
[features]
Reranking

http://localhost:8983/solr/books/query?q=subjects%3A(%22England%20--%20Fiction%22%20OR%20%22Mystery%20fiction%22)&rq=%7B!ltr%20model=linear
Model1%20reRankDocs=100%20efi.favouriteSubject=%22Mystery%20fiction%22%20efi.fromMobile=1%20efi.age=25%20efi.userLanguage=%22en%22%7D&f
l=id,title,subjects,downloads,score,[features]&debug=results
Hands On: Run a reranking query

model=myModelA model=myModelB reRankDocs=100}&fl=id,score
To obtain the model that interleaving picked for a search result, computed during reranking,
add [interleaving] to the ﬂ parameter, for example:
model=myModelA model=myModelB reRankDocs=100}&fl=id,score,
[interleaving]
Reranking with Interleaving

{
"responseHeader":{
"status":0,
"QTime":0,
"params":{
"q":"test",
"fl":"id,score,[interleaving]",
"rq":"{!ltr model=myModelA model=myModelB
reRankDocs=100}"}},
"response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs":[
{
"id":"GB18030TEST",
"score":1.0005897,
"[interleaving]":"myModelB"},
{
"id":"UTF8TEST",
"score":0.79656565,
"[interleaving]":"myModelA"}]
}}
Reranking with Interleaving

http://localhost:8983/solr/books/query?q=subjects%3A(%22England%20--%20Fiction%22%20
OR%20%22Mystery%20fiction%22)&rq={!ltr%20model=linearModel1%20model=_OriginalRanking
_%20reRankDocs=100%20efi.favouriteSubject=%22Mystery%20fiction%22%20efi.fromMobile=1
%20efi.age=25%20efi.userLanguage=%22en%22}&fl=id,title,subjects,downloads,score,[fea
tures],[interleaving]&debug=results
"response":{"numFound":2,"start":0,"maxScore":1.0005897,"docs
":[
{
"id":"GB18030TEST",
"score":1.0005897,
"[interleaving]":"_OriginalRanking_"},
{
"id":"UTF8TEST",
"score":0.79656565,
"[interleaving]":"myModel"}]
}}
Interleaving with Original Score

Future Works
http://localhost:8983/solr/techproducts/query?q=test&rq={!ltr model=myModelA model=myModelB
reRankDocs=100 interleavingAlgorithm=TeamDraft}&fl=id,score
Currently the only (and default) algorithm supported is 'TeamDraft'.
How to contribute
Do you want to contribute a new Interleaving Algorithm?
You just need to :
● implement the solr/contrib/ltr/src/java/org/apache/solr/ltr/interleaving/Interleaving.java interface in a new class
● add the new algorithm in the package: org.apache.solr.ltr.interleaving.algorithms
● add the new algorithm reference in the org.apache.solr.ltr.interleaving.Interleaving#getImplementation
Limitations
● [Distributed Search] Sharding is not supported

Online Testing Learning to Rank with Solr Interleaving

More Related Content

What's hot (20)

Similar to Online Testing Learning to Rank with Solr Interleaving (20)

More from Sease (20)

Recently uploaded (20)

Online Testing Learning to Rank with Solr Interleaving