Build your own statistical engines
2 likes1,468 views
This document discusses how to build your own statistical machine translation engines. It provides an overview of statistical machine translation compared to rule-based machine translation. It outlines the requirements, preparation steps, and DoMY (Do-Moses-Yourself) toolkit for building an SMT system. Key steps include collecting parallel corpora, preprocessing the data, training language and translation models, and evaluating the engine. Regular retraining and feedback can help improve the system over time.
Build your own statistical engines
- 1. Build Your Own Statistical Machine Translation Engines Ruben de la Fuente
- 2. About Me • 4-year degree in translation • Worked as translator for 10+ years • Working full time in MT for the past year
- 3. Agenda • Quick comparison with RbMT • Fundamentals of SMT • Requirements and preparation • Using DoMY
- 4. Disclaimer • I’m not saying SMT is better • I’m not saying SMT is right for you
- 5. Statistical Machine Translation Computer learns to translate through statistical analysis of alignment in bilingual corpora
- 6. Rule-based Machine Translation User Dictionaries + Grammar and translation rules
- 7. SMT: Pros and Cons Pros Cons Quick to build Unpredictable Cheap Quick Fluent improvements not easy
- 8. Features of an SMT system • Translation Model: table containing source and target phrases, together with a probability score (accuracy) • Language Model: list of sequences of n-words in target language together with a probability score (fluency)
- 9. Language and Translation Models • LM (fluency) • TM (accuracy)
- 10. Tokenization and recasing Breaking up text in Lowercase all words meaningul units (tokens) File > file file? > file ? file. > file . File! > file !
- 11. Requirements: Computing •4 GB RAM PC needed •Ubuntu 10.04 64-bit OS •Virtual Machine OK
- 12. Requirements: size MS Translator Hub recommends at least 10k segments I have gotten good results with 100-200k segments Roughly over 1 million words corpus
- 13. Publicly Available Corpora • Opus (ECB, EMA, OpenOffice) • Acquis Communautaire • Europarl • Hansard • Multilingual websites: Bitextor
- 14. Bitextor is Cunning www.mywebsite.com/en/overview.html www.mywebsite.com/es/overview.html <title>My source text</title> <title>My target text</title>
- 15. Requirements: relevance Data needs to be in-domain
- 16. Requirements: quality Garbage in, garbage out Diagnose your TMs with automated QA checks (e.g. glossary adherence, length)
- 21. Remove Markup Markup brings noise to the learning process Click <strong>Send</strong> Haga clic en <strong>Enviar</strong>
- 22. Do-Moses-Yourself (DoMY) Moses: state-of-the-art extensively used open source SMT toolkit DoMY: extension of Moses making installation and configuration easier
- 23. Online SMT Portals Cons letsmt.eu NDA-compliance smartmate.co Availability Speed
- 24. DoMY (Basics) Graphs: import-tmx, clean-LM/TM, build LM/TM, train, translate. Ini files: configuration (language pairs, paths for input and output). Folder structure: always include superdomain, domain and subdomain
- 25. Folder structure corpus graphs
- 26. Run from terminal Edit ini Command line
- 27. Running from GUI
- 28. Graphs Graph Function Input Output Import-tmx Extract data from Raw Corpora/sa tmx files Clean-tm Clean data Corpora/sa Corpora/re ady Build-lm Prepares training Corpora/re builds set for LM ady Build-tm Prepares training Corpora/re builds set for TM ady Train Trains MT engine Builds engines Translate Translates input Translation Translation files and produces s/in s/out tmx output
- 29. Tips for settings LM: 7-gram TM: 9-gram Aligner: Berkeley for distant languages
- 30. Troubleshoot Error message in terminal Log file in graph folder DoMT QA
- 31. Is Your Engine Good? A set is excluded from training to be used for evaluation (598 segments) From 0.5 BLEU points, engine is likely to perform well
- 32. Keep Improving Retrain the engine periodically as more translation corpus become available Gather feedback on what needs to be improved
- 33. Statistical PE • Keep a corpus of raw vs. PE • Treat them as separate language pairs • Run them thru DoMY • Create raw vs. PE engine • 2 engines: source > target, raw > PE
- 34. Questions? Speak now… Or reach me at: www.facebook.com/xlation www.wordbonds.es @rubendelafuente http://www.linkedin.com/in/rubendelafuente
Editor's Notes
- #11: Why? SMT is based in probability, calculated as # of a given token / total amount of tokens. Case and punctuation can disrupt the calculation.
- #14: To get good results with SMT, you need around 10.000 segments at least
- #21: Using Olifant from Okapi Framework
- #29: Clean data: remove too long/short, empty sentences