![Combination/Normalization
MAP scores for LT [Q2/EVAL, text only], 10,203 documents, 1,000 queries](https://image.slidesharecdn.com/combiningevidence-200712135324/85/Combining-Evidence-for-Cross-language-Information-Retrieval-45-320.jpg)
![System Combination
(WeightCombMNZ + Sum-To-One Norm)
October 2019 evaluation [Q2/EVAL] 10,203 documents, 1,000 queries](https://image.slidesharecdn.com/combiningevidence-200712135324/85/Combining-Evidence-for-Cross-language-Information-Retrieval-46-320.jpg)
![Systems Selection
MAP scores for LT [Q2/EVAL, text]
10,203 documents, 1,000 queries](https://image.slidesharecdn.com/combiningevidence-200712135324/85/Combining-Evidence-for-Cross-language-Information-Retrieval-55-320.jpg)
![Systems Selection
MAP scores for LT [Q2/EVAL, text]
10,203 documents, 1,000 queries](https://image.slidesharecdn.com/combiningevidence-200712135324/85/Combining-Evidence-for-Cross-language-Information-Retrieval-56-320.jpg)
![Systems Selection
MAP scores for LT [Q2/EVAL, text]
10,203 documents, 1,000 queries](https://image.slidesharecdn.com/combiningevidence-200712135324/85/Combining-Evidence-for-Cross-language-Information-Retrieval-57-320.jpg)
![Systems Selection
MAP scores for LT [Q2/EVAL, text]
10,203 documents, 1,000 queries](https://image.slidesharecdn.com/combiningevidence-200712135324/85/Combining-Evidence-for-Cross-language-Information-Retrieval-58-320.jpg)
![Random Selection
MAP scores for LT [Q2/EVAL, text], WeightCombMNZ with STO normalization, 10,203 documents, 1,000 queries](https://image.slidesharecdn.com/combiningevidence-200712135324/85/Combining-Evidence-for-Cross-language-Information-Retrieval-59-320.jpg)
Combining Evidence for Cross-language Information Retrieval
- 1. Combining Evidence for Cross-language Information Retrieval Petra Galuscakova petra@umd.edu University of Maryland, College Park 03/31/2020
- 2. Cross-language Information Retrieval (CLIR)
- 3. Cross-language Information Retrieval (CLIR) What are the possible side effects of Plaquenil?
- 4. Cross-language Information Retrieval (CLIR) What are the possible side effects of Plaquenil? Plaquenil side effects
- 5. Cross-language Information Retrieval (CLIR) What are the possible side effects of Plaquenil? Plaquenil side effects
- 6. Cross-language Information Retrieval (CLIR) What are the possible side effects of Plaquenil? Plaquenil side effects
- 7. Cross-language Information Retrieval (CLIR) What are the possible side effects of Plaquenil? Plaquenil side effects … troubles de l'accommodation, maladie de la rétine, régressant à l'arrêt du traitement....
- 8. Cross-language Information Retrieval (CLIR) What are the possible side effects of Plaquenil? Plaquenil side effects … troubles de l'accommodation, maladie de la rétine, régressant à l'arrêt du traitement.... … accommodation disorders, retinal disease, regressing when treatment is stopped...
- 9. - Swahili - Tagalog - Somali - Lithuanian - Bulgarian - Pashto Machine Translation for English Retrieval of Information in Any Language (MATERIAL) Demo
- 10. - Swahili - Tagalog - Somali - Lithuanian - Bulgarian - Pashto Machine Translation for English Retrieval of Information in Any Language (MATERIAL) Demo
- 11. Outline 1. How to build a CLIR system? 2. How to combine multiple systems? 3. How to pick out systems for the combination?
- 12. Outline 1. How to build a CLIR system? 2. How to combine multiple systems? 3. How to pick out systems for the combination?
- 18. Translation Plaquenil side effects Plaquenil Effets secondaires … troubles de l'accommodation, maladie de la rétine, régressant à l'arrêt du traitement.... Query Translation
- 19. Translation Plaquenil side effects Plaquenil Effets secondaires … troubles de l'accommodation, maladie de la rétine, régressant à l'arrêt du traitement.... … accommodation disorders, retinal disease, regressing when treatment is stopped... Query Translation
- 20. Translation Plaquenil side effects Plaquenil Effets secondaires … troubles de l'accommodation, maladie de la rétine, régressant à l'arrêt du traitement.... … accommodation disorders, retinal disease, regressing when treatment is stopped... Query Translation Document Translation
- 21. Translation Approaches ● Neural Machine Translation v1 trained using Marian ● Neural Machine Translation v2 trained without punctuation and casing using Marian ● Neural Machine Translation v3 trained using Sockeye ● Statistical Machine Translation trained using Moses ● Probabilistic Structured Queries (PSQ) which uses trained translation probabilities ● 1-best and N-best versions of the systems ● Trained using different data
- 22. Translation Approaches ● Neural Machine Translation v1 trained using Marian ● Neural Machine Translation v2 trained without punctuation and casing using Marian ● Neural Machine Translation v3 trained using Sockeye ● Statistical Machine Translation trained using Moses ● Probabilistic Structured Queries (PSQ) which uses trained translation probabilities ● 1-best and N-best versions of the systems ● Trained using different data
- 23. Translation Approaches ● Neural Machine Translation v1 trained using Marian ● Neural Machine Translation v2 trained without punctuation and casing using Marian ● Neural Machine Translation v3 trained using Sockeye ● Statistical Machine Translation trained using Moses ● Probabilistic Structured Queries (PSQ) which uses trained translation probabilities ● 1-best and N-best versions of the systems ● Trained using different data P (das Haus | house) = 0.5 P (der Haushalt | house) = 0.3 P (das Parlament | house) = 0.2
- 24. Translation Approaches ● Neural Machine Translation v1 trained using Marian ● Neural Machine Translation v2 trained without punctuation and casing using Marian ● Neural Machine Translation v3 trained using Sockeye ● Statistical Machine Translation trained using Moses ● Probabilistic Structured Queries (PSQ) which uses trained translation probabilities ● 1-best and N-best versions of the systems ● Trained using different data P (das Haus | house) = 0.5 P (der Haushalt | house) = 0.3 P (das Parlament | house) = 0.2
- 27. Collection Processing ● Character normalization ○ Tokenization (e.g. aren’t → aren ‘ t) ○ Lowercasing (e.g. New York →new york) ○ Punctuation removal (e.g. ex–prime minister →ex prime minister) ○ Diacritics removal (e.g. Galuščáková →Galuscakova) ○ Remove non-alphabetic characters (e.g. abc@gmail.com -> abc gmail com) ● Stemming (e.g. feeding cats → feed cat) ○ Applied during retrieval ○ Applied before MT training
- 30. CLIR Pipeline 30 Recording Speech Recognition Speech processing side effects ≠ “side effects”
- 32. Matcher ● Indri Language Model: ● Hidden Markov Model: ● Probabilistic Term Occurrence:
- 34. CLIR Pipeline 34 Recording Speech Recognition Speech processing 572 experiments available on the Development set
- 35. Outline 1. How to build a CLIR system? 2. How to combine multiple systems? 3. How to pick out systems for the combination?
- 36. Normalization and System Combination IR System 1 IR System 2 IR System 3 System CombinationQueryY IR System 4 Document List Y QueryZ QueryX Document List X Document List Z Score Normalization
- 37. Systems Combination Approaches S1 S2 S3 A 0.8 C -0.9 A 45 B 0.75 D -0.5 D 42 C 0.32 - E 30 ... ...
- 38. Systems Combination Approaches S1 S2 S3 A 0.8 C -0.9 A 45 B 0.75 D -0.5 D 42 C 0.32 - E 30 ... ...
- 39. Systems Combination Approaches S1 S2 S3 A 0.8 C -0.9 A 45 B 0.75 D -0.5 D 42 C 0.32 - E 30 ... ... 2
- 40. Systems Combination Approaches S1 S2 S3 A 0.8 C -0.9 A 45 B 0.75 D -0.5 D 42 C 0.32 - E 30 ... ...
- 41. Systems Combination Approaches S1 S2 S3 A 0.8 C -0.9 A 45 B 0.75 D -0.5 D 42 C 0.32 - E 30 ... ... 2
- 42. Systems Combination Approaches S1 S2 S3 A 0.8 C -0.9 A 45 B 0.75 D -0.5 D 42 C 0.32 - E 30 ... ... 2 W = 35 W = 22 W = 15
- 43. Normalization and System Combination IR System 1 IR System 2 IR System 3 System CombinationQueryY IR System 4 Document List Y QueryZ QueryX Document List X Document List Z Score Normalization
- 44. Score Normalization Ranks Scores Borda Counts Reciprocal Ranks Sum-To-One Min-Max Score = N - rank + 1 Where (N is number of returned documents) Score = 1 / rank
- 45. Combination/Normalization MAP scores for LT [Q2/EVAL, text only], 10,203 documents, 1,000 queries
- 46. System Combination (WeightCombMNZ + Sum-To-One Norm) October 2019 evaluation [Q2/EVAL] 10,203 documents, 1,000 queries
- 47. Outline 1. How to build a CLIR system? 2. How to combine multiple systems? 3. How to pick out systems for the combination?
- 48. System Selection ● How to pick a system combination that will work well on evaluation set? ● How to do so efficiently? ○ Thousands of possible setups ○ ~500 systems available on the Development set ○ 1020 ways to select 10 systems for combination ○ Need to reduce number of required experiments
- 51. Manual System Selection 572 systems of LT on Development set Pre-selection of 15 systems All 6-way combinations System Selection
- 52. Human Clusters System Selection 572 systems of LT on Development set Pre-selection of 15 systems All 6-way combinations System Selection Manual Best
- 53. Clustering 572 systems of LT on Development set ● Correctly retrieved Query/Document relevance pairs ● Jaccard coefficient ● Spectral clustering
- 54. Clustering 572 systems of LT on Development set ● Correctly retrieved Query/Document relevance pairs ● Jaccard coefficient ● Spectral clustering https://towardsdatascience.com/unsupervised-machine-learning-spectral-clustering-algorit hm-implemented-from-scratch-in-python-205c87271045
- 55. Systems Selection MAP scores for LT [Q2/EVAL, text] 10,203 documents, 1,000 queries
- 56. Systems Selection MAP scores for LT [Q2/EVAL, text] 10,203 documents, 1,000 queries
- 57. Systems Selection MAP scores for LT [Q2/EVAL, text] 10,203 documents, 1,000 queries
- 58. Systems Selection MAP scores for LT [Q2/EVAL, text] 10,203 documents, 1,000 queries
- 59. Random Selection MAP scores for LT [Q2/EVAL, text], WeightCombMNZ with STO normalization, 10,203 documents, 1,000 queries
- 60. Takeaways ● System combination achieves 40% improvement comparing with a single best system ● Selecting systems for a combination is crucial for effectiveness and perfomance ● Human-aided and manual methods work best ○ But fully automatic clustering-based method is robust
- 61. Other Research Directions https://www.dropbox.com/s/vkxl9awdep80sq5/shamus.mp4?dl=0
- 62. Visual Features ● Similar setting: - Feature Signatures ● Concept detection and visual object similarity ● Same faces
- 63. Video Retrieval Results C - Context M - Metadata V - Visual Features MediaEval 2014 Search and Hyperlinking 4,000 hours of BBC broadcast MAP-tol
- 64. Video Retrieval Results C - Context M - Metadata V - Visual Features MediaEval 2014 Search and Hyperlinking 4,000 hours of BBC broadcast MAP-tol