Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit - Tim Allison
0 likes256 views
This document introduces Quaerite, a search relevance toolkit for testing search relevance parameters offline. It allows running experiments to test different combinations of tokenizers, filters, scoring models and other parameters to evaluate search relevance without live user queries. The toolkit supports experimenting with all parameter permutations using grid search or random search, and also incorporates a genetic algorithm with cross-fold validation. It currently supports Apache Solr and plans to add support for ElasticSearch. The goal is to help optimize search relevance through offline testing of parameter configurations.
Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit - Tim Allison
- 1. © 2019 The MITRE Corporation. All rights reserved. Quaerite – Search Relevance Toolkit Tim Allison tallison@apache.org, @_tallison April 24, 2019 Haystack Conference Approved for Public Release; Distribution Unlimited. Case Number 18-3138-5
- 2. | 2 | © 2019 The MITRE Corporation. All rights reserved. Debt of Gratitude ▪ Thank you Doug Turnbull, John Berryman and Open Source Connections for the inspiration/examples/training with tmdb and for sharing your ground truth set!
- 3. | 3 | © 2019 The MITRE Corporation. All rights reserved. Yet Another Toolkit? Why!? ▪ How many parameters do we have? ▪ How many permutations of those parameters are available?
- 4. | 4 | © 2019 The MITRE Corporation. All rights reserved. Available Parameters ▪ 14 tokenizers https://lucene.apache.org/solr/guide/7_1/tokenizers.html ▪ ~45 token filters (not including language-specific token filters – see next slide) https://lucene.apache.org/solr/guide/7_1/filter-descriptions.html ▪ Query parsers ▪ Query operators, minimum should match, should, must, not ▪ Token/field based scoring – best_fields, most_fields, cross_fields ▪ Field boosting ▪ Phrasal boosting/shingling ▪ Synonym lists, taxonomies ▪ Similarity scoring parameters (with BM25) ▪ Elevate ▪ External signal enrichment – manual or automatic (NLP – entity extraction, categorization, etc.) ▪ Reranking via machine learning (Learning to Rank) | 4 | © 2019 The MITRE Corporation. All rights reserved. For internal MITRE use
- 5. | 5 | © 2019 The MITRE Corporation. All rights reserved. Each Token Filter Can Have Many Parameters <filter class="solr.WordDelimiterFilterFactory" protected="protwords.txt" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0" preserveOriginal="1"/> | 5 | © 2019 The MITRE Corporation. All rights reserved. For internal MITRE use
- 6. | 6 | © 2019 The MITRE Corporation. All rights reserved. Overview – Offline testing toolkit Prerequisites: 1. Reliable, generalizable ground truth 2. Reliable, useful underlying data 3. Offline metric has to have some connection to KPIs 4. Expertise – you still have to know what you’re doing!!!
- 7. | 7 | © 2019 The MITRE Corporation. All rights reserved. Main Tools 1. Run Experiments 2. Generate Experiments ▪ All permutations (grid search) ▪ Random experiments (random search) 3. Genetic Algorithm ▪ Cross-fold validation!!! ▪ Complementary to LTR -- main diff is algorithm and in running offline to tune general settings rather than as reranking top n
- 8. | 8 | © 2019 The MITRE Corporation. All rights reserved. Odds and Ends ▪ Analyzer Comparison over (mostly) the index ▪ Significant Terms (yawn…for archaic versions of Solr)…and planning to add these as parameters in “generate experiments”
- 9. | 9 | © 2019 The MITRE Corporation. All rights reserved. Adding Porter Stemming: create account creat created: 709 create: 551 creating: 269 creates: 153 creat: 1 account account: 3244 accounts: 1924 accounting: 1548 accountants: 340 accountant: 176 accounted: 134 accountability: 74 accountable: 74 accountancy: 65 account's: 7 accountant's: 7
- 10. | 10 | © 2019 The MITRE Corporation. All rights reserved. Status ▪ Alpha release 3/22/2019 (Solr only) ▪ Beta1 release this week (?) – This will include support for ElasticSearch ▪ Dream – Incorporate experiment generation/GA into Rated Ranking Evaluator (RRE) – Apache Incubator -> Top Level Project (TLP)
- 11. | 11 | © 2019 The MITRE Corporation. All rights reserved. Links ▪ Main site: https://github.com/mitre/quaerite ▪ Examples: https://github.com/mitre/quaerite/blob/master/quaerite- examples/README.md ▪ Contact – tallison@apache.org – @_tallison