Neuro-symbolic is not enough, we need neuro-*semantic*
Download as PPTX, PDF0 likes2,575 views
The document discusses the distinction between 'neuro-symbolic' and 'neuro-semantic' systems, emphasizing that many existing knowledge graph embeddings fail to capture true semantics, which is linked to predictable inference. It critiques the concept of 'wishful mnemonics' in AI terminology that misleads understanding of underlying algorithms. The author proposes methods for enhancing embeddings to retain semantic meaning in knowledge representations during AI system training.
![Make embeddings semantic again!
(Outrageaous Ideas paper at ISWC 2018)
Abstract
The original Semantic Web vision foresees to describe
entities in a way that the meaning can be interpreted both
by machines and humans. [But] embeddings describe an
entity as a numerical vector, without any semantics
attached to the dimensions. Thus, embeddings are as far
from the original Semantic Web vision as can be. In this
paper, we make a claim for semantic embeddings.
Proposal 1: A Posteriori Learning of Interpretations.
Reconstruct a human-readable interpretation from the
vector space.
Proposal 2: Pattern-based Embeddings.
Use patterns in the knowledge graph to choose
human-interpretable dimensions in the vector space.
30
Neither of these are aimed at predictable inference
-> no semantics ](https://image.slidesharecdn.com/2024genesy-240528065743-6eabade7/85/Neuro-symbolic-is-not-enough-we-need-neuro-semantic-30-320.jpg)
Neuro-symbolic is not enough, we need neuro-*semantic*
- 1. “neuro-symbolic” ≠ “neuro-semantic” Frank van Harmelen, Learning & Reasoning Group Vrije Universiteit Amsterdam Creative Commons License CC BY 3.0: Allowed to copy, redistribute remix & transform But must attribute
- 2. The K in “neuro-symbolic” stands for “knowledge” * Frank van Harmelen, Learning & Reasoning Group Vrije Universiteit Amsterdam Creative Commons License CC BY 3.0: Allowed to copy, redistribute remix & transform But must attribute 2 * With thanks to Wouter Beek
- 3. “neuro-symbolic” should be “neuro-semantic” Frank van Harmelen, Learning & Reasoning Group Vrije Universiteit Amsterdam Creative Commons License CC BY 3.0: Allowed to copy, redistribute remix & transform But must attribute
- 4. “GeNeSy” should be “GeNeSe” Frank van Harmelen, Learning & Reasoning Group Vrije Universiteit Amsterdam Creative Commons License CC BY 3.0: Allowed to copy, redistribute remix & transform But must attribute
- 5. I will illustrate my point on link prediction over knowledge graphs (= simplest form of GeNeSy) but it applies to any Generative NeSy 5 Abstracts in Scopus on “link prediction” AND “knowledge graph”: 1200+ papers
- 6. Bluffer’s Guide to KG Link Prediction (via embeddings) https://docs.ampligraph.org/ basedIn ?
- 7. Drug1 Protein2 Protein1 Drug2 binds Bluffer’s Guide to KG Embedding • Link prediction But also: • Entity classification • Query answering • …. binds
- 8. prediction algorithm 8 Clearly neuro-symbolic: Some things are better done geometrically (and not symbolically) From Symbols to Data and back again
- 9. 9 Example: prediction of unwanted side-effects in polypharmacy polypharmacy side-effects data as a knowledge graph <drug-x, interacts-with, protein-y> embedding Side effects <drug-x, polypharma-side-effect, drug-y> symbol:node drug-x model:tensor From Symbols to Vectors and back again learn
- 10. 10 But how well do these embeddings capture the intended meaning of the KG? TransE: |h+r-t| RotateE: symmetry <h,r,t> and <t,r,h> composition father’s mother = mother’s father
- 11. Claim None of the commonly used embeddings capture any semantics What is “semantics”? (this is the Semantic Web Conference, after all) 11
- 12. This is not semantics 12
- 13. It is “wishful mnemonics” 13 13
- 14. Artificial Intelligence meets natural stupidity, Drew McDermott, 1976 14
- 15. Artificial Intelligence meets natural stupidity, Drew McDermott, 1976 15 Wishful Mnemonics A major source of confusion in AI programs is the use of mnemnonics like “UNDERSTAND” or “GOAL”. If a programmer calls the main loop of their program “UNDERSTAND”, they may mislead a lot of people, most prominently themselves. What they should do instead is refer to this main loop as “G0034” and see if they can convince themselves or anyone else that G0034 implements some part of understanding. It is much harder to do this when using terms like “G0034”. When you say UNDERSTAND(x), you can just feel the
- 16. Artificial Intelligence meets natural stupidity, Drew McDermott, 1976 16 Prescription medicine for every AI researcher: In order to maintain your mental hygiene, read “Artificial Intelligence meets natural stupidity” once yearly.
- 17. So, this is “wishful mnemonics” 17 17
- 18. “wishful mnemonics” is not semantics for your computer 18 G0034 H9945 XB56B RB56 B599 K64 W87 U654 B6 7B3 86G 86G 86G 86G K64 BA21 BA51 86H
- 19. It is just a datagraph for your computer 19 G0034 H9945 XB56B RB56 B599 K64 W87 U654 B6 7B3 86G 86G 86G 86G K64 BA21 BA51 86H It is symbolic, but not semantic
- 20. It is just a datagraph for your computer 20 G0034 H9945 XB56B RB56 B599 K64 W87 U654 B6 7B3 86G 86G 86G 86G K64 BA21 BA51 86H Remember: “neuro-symbolic” should be “neuro-semantic”?
- 21. 21 “logical semantics” is also not semantics for your computer It just maps one formal system (called “syntax”) to another formal system (called “semantics”)
- 22. Frank Lynda birth-place • married-to relates person to person Lynda is person • married-to relates 1 person to 1 person Lynda = Hardman lowerbound upperbound Hardman married-to So what is semantics for your computer?
- 23. The semantics is in the Reserved Symbols RDF Schema Ontology Instance Schema Data
- 24. The semantics is in the Reserved Symbols RDF Schema Ontology Instance Schema Data
- 25. Ontology Instance Schema Data The semantics is in the Reserved Symbols RDF Schema
- 26. Claim 26 None of the commonly used embeddings capture any semantics Because none of the commonly used KG embeddings respect any of the reserved symbols from RDF Schema or OWL. Embeddings do “distributional semantics”, but predictable co-occurrence ≠ predictable inference Just like LLMs
- 27. Claim None of the commonly used embeddings capture any semantics Because none of the commonly used embeddings can represent universal quantification (and that’s where the inference comes from) Embeddings do “variable free sentences” only, and those don’t allow for any inference. 27 has-birth-place domain: person range: location
- 28. So: this is not a knowledge graph, It is a data graph because it doesn’t support any inference and therefore doesn’t have any semantics 28
- 29. But surely other people have noticed this before? 29
- 30. Make embeddings semantic again! (Outrageaous Ideas paper at ISWC 2018) Abstract The original Semantic Web vision foresees to describe entities in a way that the meaning can be interpreted both by machines and humans. [But] embeddings describe an entity as a numerical vector, without any semantics attached to the dimensions. Thus, embeddings are as far from the original Semantic Web vision as can be. In this paper, we make a claim for semantic embeddings. Proposal 1: A Posteriori Learning of Interpretations. Reconstruct a human-readable interpretation from the vector space. Proposal 2: Pattern-based Embeddings. Use patterns in the knowledge graph to choose human-interpretable dimensions in the vector space. 30 Neither of these are aimed at predictable inference -> no semantics
- 31. From TransE to TransOWL (and from TransR to TransROWL) 31 TransOWL: TransE Loss function Summed over all triples
- 32. More radical idea: use more of the geometry to capture the semantics 32 Male Father 𝐹𝑎𝑡ℎ𝑒𝑟 ⊑ 𝑀𝑎𝑙𝑒 Parent 𝐹𝑎𝑡ℎ𝑒𝑟 ⊑ 𝑃𝑎𝑟𝑒𝑛𝑡 Spheres: ELEm, EmEL++ Male Parent Father Boxes: BoxEL, Box2EL 𝑃𝑎𝑟𝑒𝑛𝑡 ⊓ 𝑀𝑎𝑙𝑒 ⊑ 𝐹𝑎𝑡ℎ𝑒𝑟 ? Kulmanov, 2019 Modal, 2021 Xiong, 2022 Jackermeier, 2024
- 33. More radical idea: use more of the geometry to capture the semantics
- 34. Almost done! • Take home 1: don’t just use symbols, make sure you use semantics • Take home 2: easy way to check for semantics: check for predictable inference • Note: GNN, GCN, RGCN, GAE are neuro-symbolic but not neuro-semantic systems Our illustration on link prediction generalises to many other neuro-symbolic neuro-semantic systems 34
- 35. Final 3 slides: Question: Where/when should the semantics play a role? Answer: anywhere in the architecture 35
- 36. During training Symbolic loss function: loss= dataloss + violation of semantics
- 38. During training Symbolic loss function: loss= dataloss + violation of semantics See survey of 100+ systems in Von Rueden et al, Learning, 2019 flower? cushion? “Parts of a chair are: cushion and armrest” “Given the context of chair, a cushion is much more likely than a flower” P(cushion|chair) >> P(flower|chair)
- 39. During inference symbolic consistency check queen crown wears 39 shower cap ? Predict Select crown? showercap? Showercap (97.5% certainty)
- 40. After inference Symbolic justification queen wears 40 shower cap ? Predict Justify Explain crown?
- 41. Takeaways Symbolic ≠ semantic Instead: semantics = predictable inference If you move from one representation to another, make sure not to lose the predictable inference Too late to rename neuro-symbolic to neuro-semantic but: If you enrich an LLM with a KG, Let it be a knowledge graph, not just a data graph
Editor's Notes
- #9: DEZE klopt toch niet? Zie eerder.
- #23: Mind-reading game to explain semantics. If I show the audience the top triple, and we share a little bit of background knowledge in the square box (“ontology”), I can predict what the audience will infer from the top-triple. The shared background knowledge forces us to believe certain things (such that the right blobs must be locations) , and forbids us to believe certain things (such as that the two right blobs are different). By increasing the background knowledge the enforced conclusions (lowerbound on agreement) and the forbidden conlusions (upperbound on agreement) get closer and closer, and the remaining space for ambiguity and misunderstanding reduces. Not only misunderstanding between people, but also between machines. Slogan: semantics is when I can predict what you will infer when I send you something.