Enabling multilingual search
Download as PPTX, PDF2 likes182 views
The document discusses the AGRIS approach to enabling multilingual search within its bibliographic database of 8 million publications in the agricultural domain, addressing the challenge of cross-language information retrieval. It outlines a multilingual query expansion module that uses the Agrovoc controlled vocabulary to enhance search capabilities and improve result accuracy. Future work aims to enhance user experience through better query navigation and result presentation.
Enabling multilingual search
- 1. Enabling Multilingual Search through Controlled Vocabularies: the AGRIS Approach Fabrizio Celli, Johannes Keizer MTSR 2016
- 2. AGRIS • Bibliographic database of 8 million multilingual publications in the food and agricultural domain • 350,000 visits/month from more than 200 countries and territories (Google Analytics) • Need to support cross-language information retrieval Enabling Multilingual Search through Controlled Vocabularies: the AGRIS Approach 2
- 3. Cross-language information retrieval • When a user queries AGRIS, results refer to the language of the query and of AGRIS metadata – the user query 稻米 returns all bibliographic references containing the word 稻米 in title, abstract, or as a keyword • But the user may be interested in results in all languages or in a subset of them! • Multilingual controlled vocabulary is a valid tool to deal with this scenario Enabling Multilingual Search through Controlled Vocabularies: the AGRIS Approach 3
- 4. Query filters are essential to reduce the number of results after multilingual query expansion Enabling Multilingual Search through Controlled Vocabularies: the AGRIS Approach 4
- 5. Multilingual query expansion module • Given a user query, the system: – Uses AGROVOC to translate keywords – Expands the query, boosting keywords provided by the user – Returns results in all available languages • The process relies on an intermediate Solr index – It contains AGRIVOC RDF – For each concept identified by a URI, the index stores preferred and alternative labels in all languages Enabling Multilingual Search through Controlled Vocabularies: the AGRIS Approach 5
- 6. Multilingual Query Expansion Module AGRIS website 1. Query AGRIS Query pattern analyzer Query expander AGROVOC label index 2. Expand the query 2.1 2.2 3. Return the expanded query Q1 AGRIS core index 4. Use Q1 to query AGRIS index 稻米 "稻米"^50 OR (“Rice" OR "चावल" OR "Reis" OR "рис" OR "ເຂົ້ າ" OR "벼" OR "Arroz" OR "Riso" OR "Riz" OR "rizs" OR "rýže" OR ""أرز OR "ข้าว" OR "米" OR "ryža" OR ""برنج OR "pirinç") 6
- 7. Analysis of results • Correctness of results depends on the correctness of the AGROVOC thesaurus and AGRIS metadata Source query English translation Number of results Number of results of multilingual search 稻米 rice 14 166,639 फसलें crops 0 474,854 latte milk 8,019 189,475 Klimaänderung climate change 23 31,028 "su muhafazası" water conservation 22 15,285 إنتظامحراريللتربة soil thermal regimes 21 368 "forest mensuration" forest mensuration 3,679 3,930 7
- 8. Performance and Usage • The execution of multilingual search requires 68.75 milliseconds more than the default search • 2% of AGRIS active users enable the multilingual search – 350,000 users/month – 80% come from Google.com and Google Scholar – 20% represent “active” users – 1,400 users/month use multilingual search Enabling Multilingual Search through Controlled Vocabularies: the AGRIS Approach 8
- 9. Synonyms Query Expansion Module • The union of preferred and alternative labels compose the set of synonyms for that language available in AGROVOC – Groundnuts: 2,824 results – Peanuts: 6,750 results • If the user searches for “Peanuts” and enables the synonyms expansion module: – 9,222 results (352 records contain both “Peanuts” and “Groundnuts” ) Enabling Multilingual Search through Controlled Vocabularies: the AGRIS Approach 9
- 10. Conclusions • AGRIS relies on a controlled vocabulary to implement multilingual search and synonyms expansion • Experimental results demonstrate significant improvements of recall in both cases • Future work: – Generalizing or restricting the topic of a query by navigating the hierarchy of AGROVOC concepts – Automatically performs different query expansions and combinations of them, presenting to end users alternative subsets of results Enabling Multilingual Search through Controlled Vocabularies: the AGRIS Approach 10
Editor's Notes
- #7: the multilingual query expansion module queries the AGROVOC label index to obtain translations of source keywords. The module matches source keywords against both preferred and alternative labels to identify the AGROVOC concept, but it considers only preferred labels as output of the translation process. In fact, alternative labels can mediate query expansion with synonyms. After that, the module expands the source query by building a union of source keywords and their translations. The system boosts source keywords by a factor of 50, since we think that it is important to return to users results of their original query first, and then results of the multilingual query.
- #8: Correctness of results depends on the correctness of the AGROVOC thesaurus and AGRIS metadata. A community of domain experts from different countries contributes to the quality and correctness of labels available in AGROVOC. Thus, the multilingual translation based on AGROVOC is highly reliable as far as the agricultural domain concerns.
- #9: This number is quite satisfying, since multilingual search is an advanced functionality and we expect a small percentage of usage. In addition to that, the multilingual search is a new AGRIS functionality and it needs time to reach the public. It is highly likely that the percentage will increase over the time and after we will promote the multilingual search in webinars and events.