Poincare embeddings for Learning Hierarchical Representations
Download as PPTX, PDF16 likes43,177 views
This document summarizes a paper that introduces Poincaré embeddings, which use hyperbolic geometry to learn hierarchical representations of data. Poincaré embeddings represent data in the Poincaré disk model of hyperbolic space, which allows hierarchical structures to be automatically captured. The paper proposes optimizing the embeddings with Riemannian stochastic gradient descent. Evaluation on word taxonomy and graph link prediction tasks shows Poincaré embeddings achieve better results than previous methods at reconstructing the underlying hierarchical structures.
Poincare embeddings for Learning Hierarchical Representations
- 1. Poincaré Embeddings for Learning Hierarchical Representations July 4, 2017 Tatsuya Shirakawa ABEJA Inc.
- 3. CONFIDENTIAL Today’s Paper Paper Stats • Guys from FAIR • Sumitted to arXiv at May 26, 2017 https://arxiv.org/abs/1705.08039 • Sumitted to NIPS2017? Key Contributions • Introducing hyperbolic geometry (Poincaré disk model) into word/graph embeddings paradigm • Automatically capture hierarchical structure of data • Achieved incredible better results than previous works.
- 4. CONFIDENTIAL 1. Problems 2. Hyperbolic Geometry 3. Poincaré Embeddings (and Some Incredible Results) Agenda 4
- 6. CONFIDENTIAL Find good representation(embedding) of items such that underlying hierarchical relation structure are well reconstructed The Problem
- 7. CONFIDENTIAL Embed nouns in WordNets such that related nouns are close in embedded space Taxonomy Embedding 7 http://www.nltk.org/book_1ed/ch02.html
- 8. CONFIDENTIAL Embed nodes in given graph such that missing links are well-reconstructed Graph Link Prediction 8 http://ml.cs.tsinghua.edu.cn/~jiaming/publications/
- 10. CONFIDENTIAL • Geometry with negative curvature • Many models (realizations): - Poincaré half space model - Poincaré disk model … each is isometric Hyperbolic Geometry 10
- 11. CONFIDENTIAL • Defined on upper half space with metric • Distance btw points is Poincaré Half Space Model 11
- 12. CONFIDENTIAL 12 Tree representation in H https://arxiv.org/abs/1006.5169 • Tree structure is well represented in Poincaré half space
- 13. CONFIDENTIAL • A realization of hyperbolic geometry • Defined on equipped with metric of • Distance btw points is Poincaré Disk Model 13 M.C. Escher's Circle Limit III, 1959
- 14. CONFIDENTIAL (for simplicity: 2-dim, identify as ) Relation to Poincaré Half Space Model 14 https://arxiv.org/abs/1006.5169
- 15. CONFIDENTIAL • Euclidean space is too narrow to embed hierarchical (tree) structures Why not Euclidean Space? 15 Surface Area / # of leaf nodes Volume / # of nodes Euclidean Ball O(R^n) O(R^n) b-ary tree O(b^R) O(b^R) ※ R=radius of ball or depth of tree
- 16. CONFIDENTIAL • b-array tree can be interpreted as discrete analogue of Poincaré disk Why Hyperbolic Space? 16
- 17. CONFIDENTIAL • Hyperbolic space is far more appropriate than Euclidean space to represent hierarchical structure • Many equivalent models - Poincaré half space model - Poincaré disk model … Conclusion Here 17
- 18. CONFIDENTIAL • R. Kleinberg, “Geographic routing using hyperbolic spaces”, 2007 • M. Boguna et al., “Sustaining the internet with hyperbolic mapping”, 2010 • P. D. Hoff et al., “Latent space approaches to social network analysis”, 2016 • A. B. Adcock et al., “Tree-like structure in large social and information networks’, 2013 • D. Krioukov et al., “Hyperbolic geometry of complex networks”, 2010 Prior Works around hyperbolic geometry applications 18
- 20. CONFIDENTIAL 1. Parametrize each item in Poincaré ball 2. Optimize them by Riemannian optimization under metric of Proposed Method
- 21. CONFIDENTIAL 1. Compute stochastic (Euclidean) gradient 2. Correct metric 3. Apply GD 4. Project onto space Riemannian SGD 21
- 22. CONFIDENTIAL Embed nouns in WordNets such that related nouns are close in embedded space Taxonomy Embedding 22 http://www.nltk.org/book_1ed/ch02.html
- 23. CONFIDENTIAL Maximize Reconstruction setting: - D is full relations Prediction setting - D is subset of full relations Objective Function 23 randomly chosen 10 negative samples
- 25. CONFIDENTIAL 25
- 26. CONFIDENTIAL Embed nodes in given graph such that missing links are well-reconstructed Graph Link Prediction 26 http://ml.cs.tsinghua.edu.cn/~jiaming/publications/
- 27. CONFIDENTIAL Minimize the cross entropy of probability Objective Function 27
- 29. CONFIDENTIAL • Poincaré embeddings automatically capture hierarchical structure from data • Riemannian SGD provides the way to optimize Poincaré embeddings • Achieved quite good results on word/graph embedding tasks Summary 29