Machine Translation Quality Estimation - A Linguist's Approach
Download as PPTX, PDF0 likes380 views
This document discusses machine translation quality estimation using linguistic features from complexity, adequacy, and fluency dimensions. It presents a linguist's approach to quality estimation, including the use of features like length, terminology, grammar errors, and automated post-editing. Results show scores aligned with post-editing time and effort for Spanish-Latin American samples. Challenges and future work are also outlined.
Machine Translation Quality Estimation - A Linguist's Approach
- 1. MACHINE TRANSLATION QUALITY ESTIMATION A Linguist’s Approach
- 2. WHAT IS MT QUALITY ESTIMATION? Automatically providing a quality indicator for machine translation output without depending on human reference translations. Our objective: Estimate quality and post-editing effort for eBay listing titles and descriptions MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 2
- 3. ONE big CHALLENGE min W Ʃ T t=1 ||(W(t)X(t) − Y (t) )||2 2 + λs||S||1 + λb||B||1,∞ subject to: W = S + B or “State-of-the-art QE explores different supervised linear or non-linear learning methods for regression or classification such as Support Vector Machines (SVM), different types of Decision Trees, Neural Networks, Elastic-Net, Gaussian Processes, Naive Bayes, among others” (Machine Translation Quality Estimation Across Domains, de Souza et al, 2014) MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 3
- 4. A LINGUIST’S APPROACH Using linguistic features from 3 dimensions: MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 4 COMPLEXITY ADEQUACY FLUENCY
- 5. FEATURES Complexity: • Length • Polysemy MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 5 Adequacy: • QA Terminology Patterns Blacklist Numbers • Automated Post-Editing • (POS) • (NER) Fluency: • Misspellings • Grammar errors
- 6. IMPLEMENTATION Checkmate+LanguageTool MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 6 Reusable Profile Detailed Report Score
- 7. TESTING • One Language (es-LA) • Short samples (~300 words) • Bigger samples (~1000 words) • Post-Edited files (~50,000 words) • pt-BR, ru-RU, zh-CN MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 7
- 8. RESULTS
- 9. MEASURING RESULTS MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 9
- 10. SAMPLES - SCORE AND TIME ALIGN MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 10
- 11. FILES - SCORE AND ED ALIGN MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 11 Average ED (es-LA, descriptions) = 72
- 12. MT QE OVER TIME MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 12
- 13. SAMPLES - OTHER LANGUAGES MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 13
- 14. CHALLENGES • False positives • Matching score and post-editing effort • Same weight for all features MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 14
- 15. WHAT’S NEXT • Tracking scores over time • Adding scores to our post-editing tool • Adding new languages • Researching new features MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 15
- 16. HOW CAN YOU USE THIS? • Tailor the model to your needs • Estimate quality at the file/segment level • Target post-editing, discard bad content • Estimate post-editing effort/time • Compare MT systems • Monitor MT system progress MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 16
- 17. Q&A THANK YOU! jrowda@ebay.com MT QUALITY ESTIMATION – A LINGUIST’S APPROACH 17
Editor's Notes
- #2: Introduction