Master Thesis of Computer Engineering: OpenTranslator
- 1. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Tesi di laurea magistrale relatore Ch.mo Prof Carlo Sansone correlatori Ing. Gabriele Piantadosi, Ph.D. Ing. Stefano Marrone candidato Giuseppe D’Onofrio Matr. M63/635 OpenTranslator: an open Machine Translation service based on Neural Networks Anno Accademico 2017/18
- 2. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Context Automated language translation Deep Learning approach for Neural Machine Translation (NMT) Contribution Training and evaluation of the state-of-the-art model (Transformer) to translate from English to Italian Building an open translation service using the Transformer model, providing: Two interfaces (web GUI and REST APIs) Free access to training data (with regular automated update)
- 3. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Machine Translation Several methods have been used: Rule-based Statistic Example-based Def: Machine Translation (MT) is the process of translating texts from one language to another with a software Neural Networks Machine Translation Neural Machine Translation Deep learning →
- 4. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica NMT architectures The standard architecture is the Encoder-Decoder Convolutional Neural Networks (CNN) Recurrent Neural Networks (RNN) with LSTM or GRU units Two main blocks Encoder: Input sentence → Internal vector representation Decoder: Internal vector representation → translated sentence RNN Encoder-Decoder by Sutskever et al. (2014) CNN ByteNet from Kalchbrenner et al. (2017)
- 5. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica The Transformer model Novelty: Multi-Head Attention to understand the context of sentences Positional Encoding: takes into account the positions of words Auto-regressive: previous generated words feed input No recurrence or convolution Encoder-Decoder architecture It is arranged in N identical layers Words are mapped into vectors through Embedding Inputs can be words or sub-words Output words are chosen through beam search Proposed by Google (2017)
- 6. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Multi-Head Attention For each iteration, Attention indicates how the model should focus on each word Computed on 8 different heads, which are linear projections in different subspaces
- 7. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Training pipeline A vocabulary of 32k sub-words has been generated Evaluation metrics: Padded cross- entropy (loss) BLEU score Corpus # Sentences # ENG Words # ITA Words ParaCrawl 8.3 M 179.1 M 180.2 M EuroParl v7 2.0 M 59.9 M 58.9 M NewsComment ary11 45.9 k 1.3 M 1.0 M OpenSubtitles 19.1 k 0.2 M 0.1 M Newstest2009 3.0 k 68 k 67 k Dataset
- 8. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Evaluation metric To score the quality of translations, BLEU score has been used BLEU counts the matching n-grams (group of n consecutive words) between a candidate translation and a reference translation, penalising translations that are shorter, longer or with different words compared to reference texts Word order is not relevant for BLEU score Example: Candidate: «The car is on the ground.» Reference: «The cat is on the mat.» Matching 1-grams 4/6 Matching 2-grams 2/5 Matching 3-grams 1/4 Matching 4-grams 0/3 log 𝐵𝐿𝐸𝑈 = min(1 − 𝑟 𝑐 , 0) + 𝑛=1 𝑁 𝑤 𝑛 𝑙𝑜𝑔 𝑝 𝑛
- 9. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Experimental Setup Training lasted 240k steps (~ 3 epochs) Batch size = 4096 Max sentence length = 150 sub-words Adam (Adaptive moment estimation) optimizer Variable learning rate with warmup Residual dropout with P = 0.2 Label smoothing GPU Hardware: NVIDIA Tesla K80 (Google Colab) with 2496 CUDA cores and 12GB RAM Tensor2Tensor: a Tensorflow implementation providing NMT models, tools to generate vocabulary, to train the model and to export it for the deployment
- 10. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Experimental Results Original Translated Confidence interval of mean (10 observations of 5k chars) in BLEU score (confidence level = 95%) Newstest OpenSubtitles Our proposal 23.32 ± 2.83 16.54 ± 2.63 Google Translate 26.45 ± 2.36 25.9 ± 2.39 DeepL 27.62 ± 2.40 19.48 ± 3.75 Microsoft Translator 24.53 ± 2.28 26.53 ± 2.58 «American stocks suffered a bloodbath on Monday and the major stock indexes registered their greatest fall in more than 20 years.» «Lunedì i titoli americani hanno subito un bagno di sangue e i principali indici dei titoli hanno registrato il calo maggiore in oltre 20 anni.» «In the past month, a lot of courses began at the University. Students from Naples and other cities are coming with their private vehicles, by train or by bus.» «Nell'ultimo mese sono iniziati numerosi corsi all'Università: studenti di Napoli e di altre città si recano con i loro veicoli privati, in treno o in autobus.» «That makes it hard for guys in my line... to get what I' d call "quality health care".» «Ciò rende difficile per le pistole nella mia linea... ottenere quello che chiamo "assistenza sanitaria di qualità".»
- 11. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Translation service architecture OpenTranslator has two interfaces: Web Rest API (POST) Input and output sentences with feedback are stored on a MySQL database. With stored data, the model can be re-trained to improve itself
- 12. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Web interface Web interface consists of two main blocks: Left: the user can write a sentence to translate Right: the result of the translation is presented The user can leave a feedback on the quality of translation Positive feedbacks are used to enrich the training dataset Training dataset and feedbacks can be freely downloaded
- 13. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Conclusions Results show comparable performances (in some cases) After a train of 7 days, BLEU score still improves The context of the training dataset strongly affects the ability to generalize The novelty of our translation service is to provide free APIs We also provided the dataset and the user feedbacks for free Future works Improve the model to achieve a better translation Add to the GUI the possibility of improve the result with the user input Deploy an automatic procedure to retrain with the users’ data Translate other languages (such as Italian to English)
- 14. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica More on RNN GRU LSTM
- 15. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Complexity analysis Layer type Complexity per layer Seq. operations Max. path length Self-Attention 𝑂 𝑛2 𝑑 𝑂(1) 𝑂(1) Recurrent 𝑂 𝑛𝑑2 𝑂(𝑛) 𝑂(𝑛) Convolutional 𝑂 𝑘𝑛𝑑2 𝑂(1) 𝑂(𝑙𝑜𝑔 𝑘 𝑛 ) Self-Attention (restricted) 𝑂 𝑟𝑛𝑑 𝑂(1) 𝑂(𝑛/𝑟)
- 16. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Positional Encoding 𝑷𝑬 𝒑𝒐𝒔+𝒌 = 𝒖 ∗ 𝑷𝑬 𝒑𝒐𝒔,𝟐𝒊 + 𝒗 ∗ 𝑷𝑬 𝒑𝒐𝒔,𝟐𝒊+𝟏 i is the dimension of sinusoid, pos is the position of the word
- 17. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Embedding
- 18. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica Beam search
- 19. OpenTranslator: an open machine translation service based on neural networks Scuola Politecnica e delle Scienze di Base Corso di Laurea Magistrale in Ingegneria Informatica BLEU 𝑝 𝑛 = 𝐶∊ 𝐶𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑠 𝑛−𝑔𝑟𝑎𝑚 ∊ 𝐶 𝐶𝑜𝑢𝑛𝑡 𝑐𝑙𝑖𝑝(n− 𝑔𝑟𝑎𝑚) 𝐶′∊ 𝐶𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑠 𝑛−𝑔𝑟𝑎𝑚′∊ 𝐶′ 𝐶𝑜𝑢𝑛𝑡(n− 𝑔𝑟𝑎𝑚′) 𝐵𝑃 = 1 𝑖𝑓 𝑐 > 𝑟 1 − 𝑒 1− 𝑟 𝑐 𝑖𝑓 𝑐 ≤ 𝑟 r = reference length, c = candidate length