New Adventures in RDF2vec
0 likes451 views
This document discusses RDF2vec, a technique for generating embeddings from knowledge graphs. RDF2vec works by generating random walks through the graph and using the walks to train an embedding model like word2vec. The embeddings produced encode similarity and relatedness of entities. Recent work has improved RDF2vec by incorporating graph structure into the walks and combining different embedding spaces. Interpretability of embeddings remains a challenge but approaches aim to learn interpretation functions or inherently interpretable patterns. RDF2vec has been applied successfully in many domains and tasks.
New Adventures in RDF2vec
- 1. 12/15/21 Heiko Paulheim 1 New Adventures in RDF2vec Heiko Paulheim University of Mannheim Heiko Paulheim
- 2. 12/15/21 Heiko Paulheim 2 Brief Introduction 2006 2008 2011 2013 2014 2017 Pre PhD Years PhD Years PostDoc Years Assistant Prof. Full Prof. SDType rdf2vec ReNewRS Kare§KoKI MELT
- 3. 12/15/21 Heiko Paulheim 3 Knowledge Graphs: At a Glance • Graph shaped knowledge representation – nodes: entities – edges: relations University of Mannheim Mannheim Baden- Württemberg Germany Heiko Paulheim DWS Group employer a f f il i a t io n part of residence s t a t e part of
- 4. 12/15/21 Heiko Paulheim 4 Knowledge Graphs in Organizations • Knowledge Graphs are used… • …in companies and organizations – collect, organize, and integrate knowledge – link isolated information sources – make information searchable and findable Masuch et al., 2016
- 5. 12/15/21 Heiko Paulheim 5 Public Knowledge Graphs • Knowledge Graphs are used… • …as (free), public resources – collect common knowledge – general purpose, not task specific – make it easy to build knowledge-intensive applications
- 6. 12/15/21 Heiko Paulheim 6 Knowledge Graphs: Out of the Dark Google’s Announcement DBpedia YAGO ResearchCyc Wikidata Freebase NELL
- 7. 12/15/21 Heiko Paulheim 7 Usage of Public Knowledge Graphs OK, Google, when will the final season of Money Heist be on Netflix? The fifth season of Money Heist will be released on September 3rd .
- 8. 12/15/21 Heiko Paulheim 8 Usage of Public Knowledge Graphs 2021-09-03 2020-04-03 release date release date has part h a s p a r t OK, Google, when will the final season Money Heist be on Netflix? . . .
- 9. 12/15/21 Heiko Paulheim 9 Usage of Public Knowledge Graphs 2021-09-03 2020-04-03 release date release date creator has part h a s p a r t cast c a s t creator c a s t Are there any other series by the same creator? creator cast cast . . . . . .
- 10. 12/15/21 Heiko Paulheim 10 Use Cases for Knowledge Graphs • Background Knowledge – e.g., company data (address, CEO, branch, …) → SAP CRM (BSc thesis 2019) – e.g., geographic regions (demographics) → for example, sales data prediction – data interpretation (e.g., Excel tables, business models) → PhD thesis under supervision • Data Integration – unified view of different data sources – relating business entities in different systems – cross-source data visualization and analytics
- 11. 12/15/21 Heiko Paulheim 11 Knowledge Graphs in Data Science • Typical cases: – predictive modeling, information retrieval, recommendation, … • For all of those, there’s sophisticated implementations – but... ?
- 12. 12/15/21 Heiko Paulheim 12 Wanted: A Bridge between Both Worlds
- 13. 12/15/21 Heiko Paulheim 13 Wanted: A Bridge between Both Worlds • Data Science tools for prediction etc. – Python, Weka, R, RapidMiner, … – Algorithms that work on vectors, not graphs • Bridges built over the past years: – FeGeLOD (Weka, 2012), RapidMiner LOD Extension (2015), Python KG Extension (2021) ?
- 14. 12/15/21 Heiko Paulheim 14 Wanted: A Bridge between Both Worlds • Transformation strategies (aka propositionalization) – e.g., types: type_horror_movie=true – e.g., data values: year=2011 – e.g., aggregates: nominations=7 ?
- 15. 12/15/21 Heiko Paulheim 15 Wanted: A Bridge between Both Worlds • Observations with simple propositionalization strategies – Even simple features (e.g., add all numbers and types) can help on many problems – More sophisticated features often bring additional improvements • Combinations of relations and individuals – e.g., movies directed by Steven Spielberg • Combinations of relations and types – e.g., movies directed by Oscar-winning directors • … – But • The search space is enormous! • Generate first, filter later does not scale well
- 16. 12/15/21 Heiko Paulheim 16 Wanted: A Bridge between Both Worlds • Excursion: word embeddings – word2vec proposed by Mikolov et al. (2013) – predict a word from its context or vice versa • Idea: similar words appear in similar contexts, like – Jobs, Wozniak, and Wayne founded Apple Computer Company in April 1976 – Google was officially founded as a company in January 2006 – usually trained on large text corpora • projection layer: embedding vectors
- 17. 12/15/21 Heiko Paulheim 17 From Word Embeddings to Graph Embeddings • Basic idea: – extract random walks from an RDF graph: Mulholland Dr. David Lynch US – feed walks into word2vec algorithm • Order of magnitude (e.g., DBpedia) – ~6M entities (“words”) – start up to 500 random walks per entity, length up to 8 → corpus of >20B tokens • Result: – node embeddings – most often outperform other propositionalization techniques director nationality
- 18. 12/15/21 Heiko Paulheim 18 A First Glance at RDF2vec Embeddings • Observation: close projection of similar entities
- 19. 12/15/21 Heiko Paulheim 19 Random vs. non-random • Maybe random walks are not such a good idea – They may give too much weight on less-known entities and facts • Strategies: – Prefer edges with more frequent predicates – Prefer nodes with higher indegree – Prefer nodes with higher PageRank – … – They may cover less-known entities and facts too little • Strategies: – The opposite of all of the above strategies • External signals (e.g., human notions of importance) – generally work better than graph-internal signals Cochez et al. (2017): Biased Graph Walks for RDF Graph Embeddings Al Taweel and Paulheim (2020): Towards Exploiting Implicit Human Feedback for Improving RDF2vec Embeddings
- 20. 12/15/21 Heiko Paulheim 20 Local Embeddings • Recap: order of magnitude (e.g., DBpedia) – ~6M entities (“words”) – start up to 500 random walks per entity, length up to 8 → corpus of >20B tokens – “Train once, reuse often” • In some cases, only a small subset (of 6M) is of interest – RDF2vec light: “train when needed” – Runtime: minutes instead of days Portisch et al. (2020): RDF2Vec Light – A Lightweight Approach for Knowledge Graph Embeddings
- 21. 12/15/21 Heiko Paulheim 21 RDF2vec: Example Applications • Data Model Matching with WebIsA and RDF2vec Portisch et al. (2019): Evaluating ontology matchers on real-world financial services data models.
- 22. 12/15/21 Heiko Paulheim 22 RDF2vec: Example Applications • Entity disambiguation: linking texts to a knowledge graph Türker et al. (2019): Knowledge-Based Short Text Categorization Using Entity and Category Embedding
- 23. 12/15/21 Heiko Paulheim 23 RDF2vec: Example Applications • Finding related research papers on CoViD-19 Steenwinckel et al. (2020): Facilitating COVID-19 Meta-analysis Through a Literature Knowledge Graph
- 24. 12/15/21 Heiko Paulheim 24 RDF2vec: Example Applications • Table search by keyword Zhang and Balog (2018): Ad Hoc Table Retrieval using Semantic Similarity.
- 25. 12/15/21 Heiko Paulheim 25 RDF2vec: Example Applications • Predicting biological interactions Sousa et al. (2021): Supervised Semantic Similarity.
- 26. 12/15/21 Heiko Paulheim 26 RDF2vec: Example Applications • Zero-Shot Image Classification Hascoet et al. (2017): Semantic Web and Zero-Shot Learning of Large Scale Visual Classes.
- 27. 12/15/21 Heiko Paulheim 27 Embeddings for Link Prediction • RDF2vec example – similar instances form clusters, direction of relation is ~stable – link prediction by analogy reasoning (Japan – Tokyo ≈ China – Beijing) Ristoski & Paulheim: RDF2vec: RDF Graph Embeddings for Data Mining. ISWC, 2016
- 28. 12/15/21 Heiko Paulheim 28 Embeddings for Link Prediction • In RDF2vec, relation preservation is a by-product • TransE (and its descendants): direct modeling – Formulates RDF embedding as an optimization problem – Find mapping of entities and relations to Rn so that • across all triples <s,p,o> Σ ||s+p-o|| is minimized • try to obtain a smaller error for existing triples than for non-existing ones Bordes et al: Translating Embeddings for Modeling Multi-relational Data. NIPS 2013. Fan et al.: Learning Embedding Representations for Knowledge Inference on Imperfect and Incomplete Repositories. WI 2016
- 29. 12/15/21 Heiko Paulheim 29 Link Prediction vs. Node Embedding • Hypothesis: – Embeddings for link prediction also cluster similar entities – Node embeddings can also be used for link prediction Portisch et al. (to appear): Knowledge Graph Embedding for Data Mining vs. Knowledge Graph Embedding for Link Prediction - Two Sides of the Same Coin?
- 30. 12/15/21 Heiko Paulheim 30 Similarity vs. Relatedness • Closest 10 entities to Angela Merkel in different vector spaces Portisch et al. (to appear): Knowledge Graph Embedding for Data Mining vs. Knowledge Graph Embedding for Link Prediction - Two Sides of the Same Coin?
- 31. 12/15/21 Heiko Paulheim 31 Similarity vs. Relatedness • (s-)RDF2vec allows an explicit trade off w/ different walk strategies Mannheim Baden- Württemberg Germany Adler Mannheim SAP Arena Reiss- Engelhorn -Museum location location location federal state country location city stadium Knowledge Graph Walk Generation Adler_Mannheim → city → Mannheim → country → Germany Adler_Mannheim → stadium → SAP_Arena → location → Mannheim SAP_Arena → location → Mannheim → country → Germany ... “Classic” RDF2vec walks city → Mannheim → country stadium → SAP_Arena → location location → Mannheim → country ... s-RDF2vec walks + RDF2vec “union walks” RDF2vec “classic” RDF2vec “edge” concatenated vector Global PCA Test Cases concatenated vector (task-specific subset) w 2 w 1 (weighted) local PCA Portisch et al. (under review): s-RDF2vec: Injecting Knowledge Graph Structure Into RDF2vec Entity Embeddings.
- 32. 12/15/21 Heiko Paulheim 32 Similarity vs. Relatedness • s-RDF2vec – using different walk strategies – combining different vector spaces (weighted combinations are possible) • 10 closest neighbors to Mannheim: Portisch et al. (under review): s-RDF2vec: Injecting Knowledge Graph Structure Into RDF2vec Entity Embeddings.
- 33. 12/15/21 Heiko Paulheim 33 Similarity vs. Relatedness • Recap word embeddings: – Jobs, Wozniak, and Wayne founded Apple Computer Company in April 1976 – Google was officially founded as a company in January 2006 • Graph walks: – Hamburg → country → Germany → leader → Angela_Merkel – Germany → leader → Angela_Merkel → birthPlace → Hamburg – Hamburg → leader → Peter_Tschentscher → residence → Hamburg Germany Angela_Merkel Hamburg birthPlace country leader Peter_Tschentscher leader residence country
- 34. 12/15/21 Heiko Paulheim 34 Similarity vs. Relatedness • Surrounding entities indicate relatedness – Hamburg → country → Germany → leader → Angela_Merkel – Germany → leader → Angela_Merkel → birthPlace → Hamburg • Same entities in similar positions indicate similarity – Germany → leader → Angela_Merkel → birthPlace → Hamburg – Hamburg → leader → Peter_Tschentscher → residence → Hamburg • Someone is a leader vs. something has a leader • Solution approach: use embedding approach that respects positions – CWINDOW / Structured Skip-ngram Portisch and Paulheim (2021): Putting RDF2vec in Order.
- 35. 12/15/21 Heiko Paulheim 35 Similarity vs. Relatedness • Why bother? – Use case: table interpretation (a special case of entity disambiguation) related similar
- 36. 12/15/21 Heiko Paulheim 36 Back to Interpretability • Hot topic: Explainable AI – Knowledge Graphs are a favorable ingredient – Human/machine interpretable knowledge → explainable systems • However: – Embeddings replace interpretable axioms with numeric vectors over non-interpretable dimensions – Where did the semantics go? Paulheim (2018): Make Embeddings Semantic Again!
- 37. 12/15/21 Heiko Paulheim 37 The 2009 Semantic Web Layer Cake
- 38. 12/15/21 Heiko Paulheim 38 The 2018 Semantic Web Layer Cake Embeddings
- 39. 12/15/21 Heiko Paulheim 39 Towards Semantic Vector Space Embeddings cartoon superhero Paulheim (2018): Make Embeddings Semantic Again!
- 40. 12/15/21 Heiko Paulheim 40 Towards Semantic Vector Space Embeddings cartoon superhero • Approach 1: learn interpretation function • Each dimension of the embedding model is a target for a separate learning problem • Learn a function to explain the dimension • E.g.: • Just an approximation used for explanations and justifications y≈−|∃character .Superhero|
- 41. 12/15/21 Heiko Paulheim 41 Towards Semantic Vector Space Embeddings cartoon superhero • Approach 2: learn inherently interpretable embeddings • Step 1: learn typical patterns that exist in a knowledge graph – e.g., graph pattern learning – e.g., Horn clauses • Step 2a: use those patterns as embedding dimensions – probably not low dimensional • Step 2b: compact the space – e.g., use dimensions for mutually exclusive patterns
- 42. 12/15/21 Heiko Paulheim 42 Towards Semantic Vector Space Embeddings • Different angle: learn interpretation for similarity function ~similar type ~same country ~connected to same entity
- 43. 12/15/21 Heiko Paulheim 43 Summary • Knowledge Graphs are a versatile ingredient for AI – Integrated view on data – Large-scale free source of background knowledge • Knowledge Graph Embeddings – Effective processing of large-scale knowledge sources – Encoding of similarity and/or relatedness • RDF2vec: explicit trade-off is possible! – Additional insights that are not explicit in the graph • aka latent semantics
- 44. 12/15/21 Heiko Paulheim 44 More on RDF2vec • Collection of – Implementations – Pre-trained models – >40 use cases in various domains
- 45. 12/15/21 Heiko Paulheim 45 Thank you! http://www.heikopaulheim.com @heikopaulheim
- 46. 12/15/21 Heiko Paulheim 46 New Adventures in RDF2vec Heiko Paulheim University of Mannheim Heiko Paulheim