Embracing diversity searching over multiple languages
0 likes1,184 views
This document discusses multi-lingual search and machine translation. It introduces Tommaso Teofili and Suneel Marthi, who work on Apache projects related to natural language processing. They discuss why multi-lingual search is important to embrace diversity online. Statistical machine translation generates translations from models trained on parallel text corpora. Phrase-based models can translate phrases as units and handle reordering better than word-based models. Apache Joshua is an open source machine translation decoder used by many organizations.
Embracing diversity searching over multiple languages
- 1. Embracing Diversity: Searching over multiple languages Tommaso Teofili Suneel Marthi June 12, 2017 Berlin Buzzwords, Berlin, Germany 1
- 2. $WhoAreWe Tommaso Teofili @tteofili Software Engineer, Adobe Systems Member of Apache Software Foundation, PMC Chair, Apache Lucene Committer and PMC on Apache Joshua, Apache OpenNLP, Apache JackRabbit Suneel Marthi @suneelmarthi Principal Software Engineer, Office of Technology, Red Hat Member of Apache Software Foundation Committer and PMC on Apache Mahout, Apache OpenNLP, Apache Streams 2
- 3. Agenda What is Multi-Lingual Search ? Why Multi-Lingual Search ? What is Statistical Machine Translation ? Overview of Apache Joshua Dataflow Pipeline Demo 3
- 4. What is Multi-Lingual Search ? 4
- 5. Searching over content written in different languages with users speaking different languages both Parallel corpora Translating queries Translating documents 5
- 6. Why Multi-Lingual Search ? 6
- 7. Embracing diversity Most online tech content is in English Wikipedia dumps: en: 62GB de: 17GB it: 10GB Good number of non-English speaking users A lot of search queries are composed in English Preferable to retrieve search results in native language … or even to consolidate all results in one language 7
- 8. UC1 — tech domain, native first 8
- 9. UC2 — native only ? 9
- 10. What is Machine Translation ? 10
- 11. Generate Translations from Statistical Models trained on Bilingual Corpora. Translation happens per a probability distribution p(e/f) E = string in the target language (English) F = string in the source language (Spanish) e~ = argmax p(e/f) = argmax p(f/e) * p(e) e~ = best translation, the one with highest probability 11
- 13. How to translate a word → lookup in dictionary Gebäude — building, house, tower. Multiple translations some more frequent than others for instance: house and building most common 13
- 14. Look at a parallel corpus (German text along with English translation) Translation of Gebäude Count Probability house 5.28 billion 0.51 building 4.16 billion 0.402 tower 9.28 million 0.09 14
- 15. Alignment In a parallel text (or when we translate), we align words in one language with the word in the other Das Gebäude ist hoch ↓ ↓ ↓ ↓ the building is high Word positions are numbered 1—4 15
- 16. Alignment Function Define the Alignment with an Alignment Function Mapping an English target word at position i to a German source word at position j with a function a : i → j Example a : {1 → 1, 2 → 2, 3 → 3, 4 → 4} 16
- 17. One-to-Many Translation A source word could translate into multiple target words Das ist ein Hochhaus ↓ ↓ ↓ ↙ ↓ ↘ This is a high rise building 17
- 19. Alignment Function Word-Based Models translate words as atomic units Phrase-Based Models translate phrases as atomic units Advantages: many-to-many translation can handle non- compositional phrases use of local context in translation the more data, the longer phrases can be learned “Standard Model”, used by Google Translate and others 19
- 20. Phrase-Based Model Berlin ist ein herausragendes Kunst- und Kulturzentrum . ↓ ↓ ↓ ↓ ↓ ↓ Berlin is an outstanding Art and cultural center . Foreign input is segmented in phrases Each phrase is translated into English Phrases are reordered 20
- 21. Decoding 21
- 22. We have a mathematical model for translation p(e|f) Task of decoding: find the translation ebest with highest probability Two types of error the most probable translation is bad → fix the model search does not find the most probable translation → fix the search ebest = argmax p(e|f) 22
- 23. Translation Process Translate this query from German into English er trinkt ja noch nichts er ↓ he Pick and input phrase, translate 23
- 24. Translation Process Translate this query from German into English er trinkt ja noch nichts er ja noch nichts ↓ he does not yet Pick and input phrase, translate 24
- 25. Translation Process Translate this query from German into English er trinkt ja noch nichts er trinkt ja noch nichts ↓ he does not yet drink Pick and input phrase, translate 25
- 26. Apache Joshua 26
- 27. Statistical Machine Translation Decoder for phrase-based and hierarch machine translation Written in Java Provide 64 language packs for machine translation https://cwiki.apache.org/confluence/display/JOSHUA/Language+P Project initiated by Johns Hopkins Univ. and University of Pennsylvania Presently incubating at Apache Software Foundation Used extensively by Amazon.com, NASA JPL https://cwiki.apache.org/confluence/display/JOSHUA @ApacheJoshua 27
- 28. Flows 28
- 29. References Apache Joshua — https://cwiki.apache.org/confluence/display/JOSHUA Apache OpenNLP — https://opennlp.apache.org GitHub — https://github.com/smarthi/BBuzz-multilang-search Slides — https://smarthi.github.io/bbuzz17-embracing-diversity-searching- over-multiple-languages/#/ 29
- 30. Credits Joern Kottmann — PMC Chair, Apache OpenNLP Matt Post — PMC Chair, Apache Joshua Bruno P. Kinoshita — Committer on Apache OpenNLP, committer and PMC on Apache Commons and Apache Jena 30
- 31. Questions ??? 31