TAUS USER CONFERENCE 2010, More data equals better machine translation – the Microsoft view
2 likes342 views
The document discusses a presentation given at the TAUS User Conference 2010 by Chris Wendt of Microsoft Research on machine translation technologies. The presentation covered Microsoft's machine translation service, which translates 32 languages using statistical machine translation models trained on vast amounts of publicly and privately sourced parallel text data. It also discussed efforts to improve translation quality through continued expansion of training data and collaborative frameworks to incorporate human feedback.
TAUS USER CONFERENCE 2010, More data equals better machine translation – the Microsoft view
- 1. TAUS USER CONFERENCE 2010 LANGUAGE BUSINESS INNOVATION 4 – 6 OCTOBER / PORTLAND (OR), USA TUESDAY 5 OCTOBER / 10.30 MORE DATA EQUALS BETTER MACHINE TRANSLATION – THE MICROSOFT VIEW Chris Wendt, Microsoft Research
- 2. Agenda Microsoft Translator At A Glance Some basic technical info Drivers of coverage and quality
- 4. Office 2010
- 6. Bing Instant Answer and Web Page Viewer
- 7. Microsoft’s MT Service at a Glance Translates 32 languages, any to any. http://translator.bing.com (text and URLs) Office: selection or whole document via research pane, ribbon and mini translator IE8 accelerator: popup, whole page Messenger translation bot Unique side-by-side web page viewer In-place translation widget with collaborative translations Bing “Translate this page” Bing “Translate This” instant answer Free API and Collaborative Translations Framework
- 8. Free Web API SOAP http://api.microsofttranslator.com AJAX http (REST) Very simple methods string input = "My input sentence."; Detect() string output = s.Translate(_appId, input, "", "de"); Translate() AddTranslation() GetTranslations() Advanced methods Array functionality for the above Text to Speech Sentence Breaking More language related methods in the works
- 10. Statistical MT - The Simple View User Input Text, web pages, Chat etc Government data Microsoft manuals Collect and store Dictionaries Train statistical Translation parallel and target Phrasebooks models Engine Translation language data Publisher data Engine Distributed Runtime Cosmos Translation APIs and UX Cluster HPC/MPI Cluster Web data Translated Output
- 11. Microsoft’s Statistical MT Engine Languages with source Syntactically informed SMT parser: English, Spanish, Japanese, French, German, Italian Source language Syntactic tree based decoder parser Rule-based post HTML handling processing Sentence breaking Case restoration Source language Surface string based decoder word breaker Distance and Contextual Syntactic Other source languages word-based translation reordering reordering model model Target Syntactic word Models language insertion and model deletion model
- 13. Data Sources Web data gathering Web-scale algorithms to find parallel pages Page and sentence alignment Existing (mostly) parallel data Microsoft manuals Dictionaries, phrasebooks Government Data Data sharing associations Linguistic Data Consortium, Taus Data Association, ELRA, … Licensed data Microsoft Press, … Comparable (non-parallel) data Wikipedia News articles
- 14. Apr-08 May-08 Jun-08 Jul-08 Aug-08 Sep-08 Oct-08 Nov-08 Dec-08 Jan-09 Feb-09 Parallel Sentences Mar-09 Apr-09 May-09 Jun-09 Jul-09 Aug-09 Sep-09 Oct-09 Nov-09 Dec-09 Jan-10 Feb-10
- 15. Human Evaluations Absolute 3 to 5 independent human evaluators are asked to rank translation quality for 250 sentences on a scale of 1 to 4 Comparing to human translated sentence No source language knowledge required 4 Ideal Grammatically correct, all information included 3 Acceptable Not perfect, but definitely comprehensible, and with accurate transfer of all important information 2 Possibly Acceptable May be interpretable given context/time, some information transferred accurately 1 Unacceptable Absolutely not comprehensible and/or little or not information transferred accurately Also: Relative evals, against a competitor, or a previous version of ourselves
- 16. Human Evaluation Scores ARA-ENU 2.81 ENU-ARA 2.13 BGR-ENU 2.70 ENU-BGR 2.17 CHS-ENU 2.54 ENU-CHS 2.38 CSY-ENU 2.27 ENU-CSY 2.01 DAN-ENU 2.84 ENU-DAN 2.58 DEU-ENU 3.17 ENU-DEU 2.63 ELL-ENU 2.65 ENU-ELL 2.12 ESN-ENU 2.80 ENU-ESN 2.69 FIN-ENU 2.46 ENU-FIN 2.26 FRA-ENU 2.67 ENU-FRA 2.44 HEB-ENU 2.53 ENU-HEB 2.37 ITA-ENU 2.83 ENU-ITA 2.35 JPN-ENU 2.53 ENU-JPN 2.52 KOR-ENU 2.35 ENU-KOR 2.59 NLD-ENU 2.55 ENU-NLD 2.47 PLK-ENU 2.67 ENU-PLK 1.66 PTB-ENU 2.79 ENU-PTB 2.51 RUS-ENU 2.72 ENU-RUS 2.20 SVE-ENU 2.83 ENU-SVE 2.40 THA-ENU 2.18 ENU-THA 2.24 TRK-ENU 2.07 ENU-TRK 2.18
- 17. Quality improvements in 2009 BLEU by Release (EX) BLEU by Release (XE) ARA BGR CHS CSY DAN DEU ELL ESN FIN FRA HEB ITA JPN KOR NLD PLK Aug-08 Jul-08 Aug-09 Jun-08 Dec-08 Oct-08 Jan-09 Jun-09 Jul-09 Apr-08 Sep-08 Nov-08 Feb-09 Oct-09 Dec-09 May-08 Apr-09 Sep-09 Nov-09 Jan-10 Feb-10 Mar-09 May-09 PTB RUS SVE THA 5.4 5.5 5.6 6.0 5.4 5.5 5.6 6.0
- 18. Experiment Results, measured in BLEU Chinese Test Set System Size System Description General Microsoft Sybase 1 8.3M General domain 14.26 29.74 34.81 2a 2.6M Microsoft 12.32 34.65 29.95 2b 2.8M Microsoft with Sybase 12.16 34.66 30.24 3a 11.5M General and Microsoft and TAUS 15.38 35.80 44.49 3b 11.5M System 3a with Sybase lambda 12.57 29.51 47.16 German Test Set System Size System Description General Microsoft Sybase 1 4.4M General Domain 25.19 40.61 34.85 2a 7.6M Microsoft 21.95 52.39 41.55 2b 7.8M Microsoft with Sybase 22.83 52.07 42.07 3a 11.1M General and Microsoft and TAUS 23.86 52.72 48.83 3b 11.1M System 3a with Sybase lambda 19.44 37.27 50.85 18
- 20. HAT: A Paradigm Shift Computer Aided Translation is becoming Human Aided Translation Machine Translation is Good enough to get the meaning across Not good enough to fully substitute human translation Merge MT with Human Translation using massive amounts of parallel data, and the community of humans
- 21. Collaborative Translation Framework Community enhanced MT Machine Translation Models Training Worldwide Translation Memory
- 22. Example: CSS Knowledge Base – Czech Data from Kai Gehrlach, Martine Smets, and Chris Moore
- 23. Search Engine Optimization Machine Translating the Czech Knowledge Base October 2009 January 2010 2.5% of content of the English 2.5% of content of the English KB is human translated to KB is human translated to Czech, ranked by page view. Czech, ranked by page view. The top 2.5% cover an estimated The top 2.5% cover an estimated 50% of the page views. 50% of the page views. The remaining content is The remaining content is untranslated. machine translated, starting December 5 and completed over the next 10 days.
- 24. Referrals from the Czech Republic 140,000 Referrals to the CSS KB site from the top 2 120,000 search engines in the Czech Republic 100,000 (google.cz and seznam.cz 80,000 to the Czech KB (blue) cs to the KB in other 60,000 All other languages languages (green) 40,000 20,000 0 Oct FY10 Nov FY10 Dec FY10 Jan FY10
- 25. Resolution Rate Across Languages Arabic Chinese (People's Republic of China) Chinese (Taiwan) Czech French German Italian Resolution rate HT Japanese Resolution rate MT Korean Portuguese Portuguese (Brazil) Russian Spanish Turkish 0% 10% 20% 30% 40% 50% 60% 70% Source: Martine Smets, Microsoft Customer Support
- 27. Adding Domain Specificity Syntactic tree based decoder Domain Custom Model Generic Other Models Language Target Model Contextual language translation model model Models This model includes The target language models parallel data for the have an effect only if there is Weight distribution domain as well as my matching data in the translation determined by Λ Training company model 27
- 28. Microsoft Translator Runtime Determines the best (85) alternative (14) Returns result to Distributer Model Leaf Leaf Leaf Leaf Server (4) Model Distributor Leaf Reassembles result Leaf Leaf Leaf Server chunk Finds an engine to translate the sentence Model Load Balancer Distributor Leaf Leaf Leaf Consults models Leaf Server Breaks chunk into Gets a chunk to Model Distributor sentences Leaf Leaf Leaf Leaf translate Server Model Distributor Leaf Leaf Leaf Leaf Server Model Leaf Leaf Leaf Leaf Server
- 29. Training 400-CPU CCS/HPC cluster Parallel Source language Data parsing Discrim. Train Model model weights weights Treelet + Source/Target Word alignment Syntactic structure word breaking extraction Target language monolingual data Language Surface Phrase table Treelet table Syntactic models model reordering extraction extraction training training training Case Target Distance and Contextual Syntactic Syntactic word restoration language word-based translation reordering insertion and model Target model reordering models model deletion model language Target model language model 29
- 30. References Chris Quirk, Arul Menezes, and Colin Cherry, Dependency Treelet Translation: Syntactically Informed Phrasal SMT, in Proceedings of ACL, Association for Computational Linguistics, June 2005 Microsoft Translator: www.microsofttranslator.com TAUS Data Association: www.tausdata.org 30