Lessons Learned.pptx
- 1. NMT Training Project Lessons Learned By Jiaqian Wang, Jiaying Lai, Lingge Liu, Qifang Liang
- 2. Agenda Project background, the overall process and results – Jiaqian What went well – Qifang What didn’t go so well – Jiaying Possible Improvements & insights for future projects – Lingge Executions plan Financial issues
- 3. Introduction • Client: BBC • Purpose: train an NMT engine for documentary subtitle (Big Cats II) • Language: en-US → zh-CN • MT training system: MS Custom Translator, Systran • Initial Goal: • Efficiency: PEMT 30% faster than human translation • Cost: PEMT 30% savings over human translation • Quality: PEMT with an acceptable score (≤20 points and zero critical error) in a sample of 1000 words
- 4. Data composition Microsoft Bleu Systran Bleu Round 1 Planet Earth I, The Hunt, Big Cat Tales; Glossary; Big Cats Episode 3 (Tuning Data), Big Cats Episode 1&2 (Test Data) 11.29 25.88 Round 2 Added Blue Planet II to the training dataset (2576 segments) 12.02 25.81 Round 3 Added Planet Earth II to the training dataset (2709 segments) 13.35 25.18 Round 4 Switch Big Cat Tales from training to tuning dataset (3196 segments) 13.76 25.24 Round 5 Removed Planet Earth I & II, and added the Ted talk TM to the training dataset (9136 segments) 9.92 24.94 Round 6 Added the official translation of Animal I to the training dataset (1249 segments) 13.59 24.9 Round 7 Added the official translation of Animal II to the training dataset 14.38 24.77 Process and Results
- 5. Time and cost Estimated time and costs Time (hrs) Cost ($) File Prep 12 480 Glossary creation 3 120 MT training 20 800 PEMT 4 108 Human Evaluation 4 200 PM fee 8 320 Total 51 2028 Actual time and costs Time (hrs) Cost ($) File Prep 12 480 Glossary creation 1 40 MT training 14 560 PEMT 4 108 Human Evaluation 4 200 PM fee 8 320 Total 43 1708
- 6. Human Translation PEMT Speed 300 words/hour 750 words/hour Rate $0.15/word $0.09/word Time 3.3 hours 1.3 hours Time savings: ~61% Cost $150 $90 Cost savings: ~40% The following data is based on a sample of 1000 words. Goals check: Time and Cost savings (30% ✔) Goals check: Quality (≤20 points and zero critical error ×) MS R1 MS R7 Systran R1 Systran R7 Minor (1) 38 24 29 25 Major (5) 24 10 9 22 Critical (10) 6 13 4 2 Total score 201 214 114 157
- 8. We kept a positive attitude. We were optimistic despite low score - Detailed adjustments we made that helped improve the score: During training Added Animal official subtitle
- 9. Effective communication & team cooperation
- 10. What did not go so well During File Prep During Training
- 11. During File Prep • Failure to get high quality data Accepted: srt./ass. format Not accepted: idx./sub. format Conversion by Subtitle Edit???
- 12. During File Prep (& PEMT) • Segments out of order (using Memsource) NMT → During PEMT / When assessing quality,
- 13. During training • Low Bleu score 1. Segments fragmented and out of order Ted? Bleu score dropped significantly Reasons: 1. Irrelevant content 2. Low quality At these high altitudes, the thin air is taking its toll…
- 14. • Low Bleu score 3. Lack of metrics conversion
- 15. Improvements & insights for future projects NMT feasibility Training data quality Management
- 16. Improvements & insights for future projects • NMT feasibility check • Is it MT friendly? • Contain cognitive/emotional/abstract/literary concepts? • Cost
- 17. Improvements & insights for future projects • Training data quality check • Translation quality • File format This Photo by Unknown Author is licensed under CC BY-NC
- 18. Improvements & insights for future projects • Management • Training process education • Unified setting standards • Unified QA check standards
- 19. Improvements & insights for future projects • Miscellaneous • Organized resource file • Take immediate actions
Editor's Notes
- #9: during training, despite having relatively low blue scores, we kept a positive attitude. To be more specific, as you can see, around round 5, the blue score lowered significantly. We quickly found out that its because of new dataset we added in during that round so we made adjustments accordingly. We switch the big cat tale data from training to tuning since round 4 which helped. In round 6 and 7, we added a new series called animal, which has an official translated subtitle. We did see result in these changes and rond 7 has the higest blue score of all rounds.
- #10: Our team also cooperated effectively on coming up with a QA table for PEMT. under error type and severity is a drop-down menu. after selecting from the menu, the yellow table on the bottom automatically calculates the score for each error type with minor error being 1, major error being 5, and critical error being 10. the green table on the right calculates how many errors there are based on the severity. we also communicated effectively and decided on a set of standards including what constitutes as a minor, major, and critical error