NABU - Multilingual Graph-based Neural RDF Verbalizer
- 1. NABU – Multilingual Graph-based Neural RDF Verbalizer Diego Moussallem1 Dwaraknath Gnaneshwar2 Thiago Castro Ferreira3 Axel-Cyrille Ngonga Ngomo1 1Data Science Group, Paderborn University, Germany 2DL group, Manipal Institute of Technology, India 3Linguistics Department, Federal University of Minas Gerais, Brazil Moussallem et al. (DICE) NABU 1 / 21
- 2. Introduction More than 2.5 exabytes per day Moussallem et al. (DICE) NABU 2 / 21
- 3. Introduction How can we generate text from Knowledge Graphs (KG)? 1 1 Taken from urlrb.gy/jqrry2 Moussallem et al. (DICE) NABU 3 / 21
- 4. Motivation Natural Language Generation (NLG) task Albert Einstein was a scientist who worked in physics. He was born in Ulm and graduated from the University of Zurich. Also, he died in Princeton and had under his guidance Ernst Gabor Straus. Scientist rdf:type dbo:almaMaterdbo:doctoralStudent dbo:field Physics dbo:birthPlace dbo:deathPlace Moussallem et al. (DICE) NABU 4 / 21
- 5. Motivation Natural Language Generation (NLG) task Albert Einstein was a scientist who worked in physics. He was born in Ulm and graduated from the University of Zurich. Also, he died in Princeton and had under his guidance Ernst Gabor Straus. Scientist rdf:type dbo:almaMaterdbo:doctoralStudent dbo:field Physics dbo:birthPlace dbo:deathPlace Current Drawback 1 English is the only widely targeted language 2 Graph structure is not fully exploited 3 Fluency is still low for sub-graphs Moussallem et al. (DICE) NABU 4 / 21
- 6. Research Question Goal Devise a natural language generation approach which takes into account the graph structure while generating text in more than one language Moussallem et al. (DICE) NABU 5 / 21
- 7. Research Question Goal Devise a natural language generation approach which takes into account the graph structure while generating text in more than one language Answer NABU Moussallem et al. (DICE) NABU 5 / 21
- 8. NABU - Multilingual RDF Verbalizer Architecture NABU consists of an encoder-decoder architecture which relies on Graph Attention Network (GAT) as encoder and Transformer as decoder. layer layer layer layer layer layer Albert Einstein was a scientist born in 1879 in Germany. Albert Einstein war ein 1879 in Deutschland geborener Wissenschaftler. Альберт Эйнштейн - ученый, родился в 1879 году в Германии. type Albert Einstein A0 birthPlace birthDate A0 A0 Germany A1 1879 A1 Scientist A1 Encoder GAT Decoder Transformer English German Russian RDF Graph OUTPUTINPUT Moussallem et al. (DICE) NABU 6 / 21
- 9. NABU - Multilingual RDF Verbalizer Encoding phase Albert Einstein type 1879 birthDate Germany Scientist type Albert Einstein A0 birthPlace birthDate A0 A0 Germany A1 1879 A1 Scientist A1 birthPlace reification 1 To avoid parameters explosion phenomenon (edges) 2 Concept from hyper graphs 3 To not be confused with RDF reification Moussallem et al. (DICE) NABU 7 / 21
- 10. NABU - Multilingual RDF Verbalizer Why GAT? Moussallem et al. (DICE) NABU 8 / 21
- 11. NABU - Multilingual RDF Verbalizer Single forward pass in NABU Source Vector Destination Vector Node Vector Label Vector Node Role layer Input vector Encoder Decoder Moussallem et al. (DICE) NABU 9 / 21
- 12. Evaluation Goals Answer the following research questions Q1 : How does our multilingual approach compare with state-of-the-art results in English? Q2 : Is NABU able to generate bilingual text while modelling two languages from distinct families? Q3 : How accurate are the multilingual texts generated by NABU? Steps Our evaluation is three-fold: 1 Monolingual 2 Bilingual 3 Multilingual Moussallem et al. (DICE) NABU 10 / 21
- 13. NABU - Evaluation Experimental Setup Model: 8-headed multi-head attention mechanism batch size - 32, Adam optimizer 0.001 Embeddings and Hidden Layers 256 dropout 0.3 Sentence length 50 and BPE 32k beam size 5 Unsupervised Tokenizer - SentencePiece WebNLG Datasets English – 9,674 RDF sets (1-7 triples), 25,298 texts, 15 domains German – 7,812 RDF sets (1-7 triples), 20,370 texts, 15 domains Russian – 5,185 RDF sets (1-7 triples), 20,800 texts, 9 domains Moussallem et al. (DICE) NABU 11 / 21
- 14. NABU - Evaluation Monolingual task Goal Measure the overall quality of NABU against current SOTA Baseline: 1 Related works (English) 2 Vanilla Transformer (German) 3 Vanilla Transformer (Russian) Benchmark: 1 WebNLG-English 2 WebNLG-German 3 WebNLG-Russian Metrics: BLEU, METEOR, chrF++ Moussallem et al. (DICE) NABU 12 / 21
- 15. NABU - Evaluation Bilingual task Goal Measure the overall quality of NABU on language families Baseline: 1 Vanilla Transformer Benchmark: 1 WebNLG-English+WebNLG-German 2 WebNLG-English+WebNLG-Russian Metrics: BLEU, METEOR, chrF++ Moussallem et al. (DICE) NABU 13 / 21
- 16. NABU - Evaluation Multilingual task Goal Measure the overall quality of NABU on English, German and Russian Baseline: 1 Vanilla Transformer Benchmark: 1 WebNLG-English+WebNLG-German + WebNLG-Russian Metrics: BLEU, METEOR, chrF++ Moussallem et al. (DICE) NABU 14 / 21
- 17. NABU - Results Monolingual Results - English Q1 : How does our multilingual approach compare with state-of-the-art results in English? NABU outperforms related works on English Model BLEU METEOR chrF++ UPF-FORGe 38.65 39.00 - Melbourne 45.13 37.00 - Moryossef et al., 2019) 47.40 39.00 - Castro et al. (2019) 51.68 32.00 - NABUGAT−Trans 66.21 41.11 71.98 Moussallem et al. (DICE) NABU 15 / 21
- 18. NABU - Results Monolingual Results Q1 : How does our multilingual approach compare with state-of-the-art results on German and Russian? NABU presents good results on German and Russian separately Models Language BLEU METEOR chrF++ Monolingual Transformerbaseline ENG 54.96 38.43 69.11 GER 50.07 34.51 63.48 RUS 46.42 27.74 56.80 NABUGAT−Trans ENG 66.21 41.47 71.98 GER 53.08 37.42 64.57 RUS 46.86 28.84 58.37 Moussallem et al. (DICE) NABU 16 / 21
- 19. NABU - Results Bilingual Results Q2 : Is NABU able to generate bilingual text while modelling two languages from distinct families? NABU generates good bilingual results Models Language BLEU METEOR chrF++ Bilingual Transformerbaseline ENG-GER 58.30 36.46 66.72 NABUGAT−Trans ENG-GER 61.99 39.51 69.68 Transformerbaseline ENG-RUS 55.30 37.90 61.63 NABUGAT−Trans ENG-RUS 49.15 33.41 64.00 Moussallem et al. (DICE) NABU 17 / 21
- 20. NABU - Results Multilingual Results Q3 : How accurate are the multilingual texts generated by NABU? NABU presents good multilingual results Models Language BLEU METEOR chrF++ Multilingual Transformerbaseline ENG-GER-RUS 53.39 36.86 60.72 NABUGAT−Trans ENG-GER-RUS 56.04 38.34 62.04 Moussallem et al. (DICE) NABU 18 / 21
- 21. Error Analysis and Discussion 1 Consistent results across languages 2 Possessive is still a source of error 3 Similar predicates with similar subjects fail to verbalize (co-reference) Albert Einstein deathPlace USA Michael Jackson deathPlace USA Albert Einstein birthPlace Ulm Michael Jackson birthPlace Gary 4 Improved generalisation (unseen entities) on all languages 5 Unseen entities on Russian were worse than on German and English Moussallem et al. (DICE) NABU 19 / 21
- 22. Summary and Future Work Summary: 66.21 BLEU on English (new SOTA) 56.04 BLEU on Multilingual settings Consistent results across languages What Next? Application of sub-graphs Exploit other NN architectures Investigate different KG Embeddings Extend to other languages Moussallem et al. (DICE) NABU 20 / 21
- 23. Thank you for your Attention! Diego Moussallem diego.moussallem@upb.de https://go.upb.de/moussallem Follow us on Twitter @DiceResearch Follow me on Twitter @diegomoussallem ⇒ Github: https://github.com/dice-group ⇒ Website: https://dice.cs.uni-paderborn.de Moussallem et al. (DICE) NABU 21 / 21