Introduction to Graph neural networks @ Vienna Deep Learning meetup
- 1. Graph Neural Networks Liad Magen March October 2020
- 3. Prof. Dr. Stefan Thurner
- 5. Seven Bridges of Königsberg → →
- 6. Location & Street representation
- 9. Paper citations as a graph
- 10. Mesh & Point cloud as a graph
- 14. Process other (non-Euclidean) formats of data
- 15. Google PageRank
- 16. Inspired usages
- 19. Graph Neural Networks ICLR Submissions Source: Twitter - @prlz77
- 20. Back to basics
- 21. Graphs, Mathematically G = (Vetrices, Edges) V = {v1 … vn) E = {(vi, vj), … }
- 22. Graph - algebraic representation G = (Vetrices, Edges) Adjacency matrix 1 - if {vi, vj} ∈ E && i != j Aij 0 - Otherwise
- 23. Graph - algebraic representation G = (Vetrices, Edges) Laplacian matrix * L = D-A Degree Adjacency Laplacian * For undirected graphs
- 24. GCN - Algorithm
- 25. GCN GCN Layer: Y = ReLU(AXW) ...but with message passing
- 26. Message Passing For each node : 1. Aggregate neighbor nodes into an intermediate representation 2. Transform the aggregated representation with a linear projection followed by a non-linearity (ReLU) Mathematically: H →Network Layer W→Network Weights A →Adjacency Matrix D →Degree Matrix
- 27. Model types ● Graph classification ○ Chemical properties of a molecule ○ Comparing user preferences / activities ● Node classification - node label prediction ○ Malicious users in a social network ○ Visually inferred Named Entity Recognition (NER) ○ Node clustering ● Edge prediction ○ Recommendation system ○ Protein-protein interaction ○ “Friend” suggestion
- 28. Frameworks
- 29. NetworkX ● Store and mutate Graphs ● Graph algorithms (Shortest path - Dijkstra, TreeWidth, clustering, centrality) ● Network analysis ● Node / edge data ● Visualization tools
- 30. DGL ● Building blocks ● Great tutorials ● Generative graphs ● Great for research and complicated tasks
- 31. ● An extension library for pyTorch ● Officially part of the pyTorch ecosystem ● Easily extensible ● Papers are implemented directly in it ● Looooooooooooooong list of ready-to-use methods and algorithms: ○ TransformerConf (2020) ○ GCN2Conv (2020) ○ DeeperGCN (2020) ○ Top-K Pooling ○ PairNorm PyTorch Geometric
- 32. Model types ● Graph classification ○ Chemical properties of a molecule ○ Comparing user preferences / activities ● Node classification - node label prediction ○ Malicious users in a social network ○ Visually inferred Named Entity Recognition (NER) ○ Node clustering ● Edge prediction ○ Recommendation system ○ Protein-protein interaction ○ “Friend” suggestion
- 34. PyTorch Geometric Example ● DataLoader ● GCN Layers
- 35. Potential Pitfalls when going ‘deep’ ● Vanishing Gradient ● Overfitting ● Over-smoothing Node-vectors become too similar ● Bottleneck (Over-Squashing) A single node vector contains data of too many nodes https://arxiv.org/abs/2006.07107 - Effective Training Strategies for Deep Graph Neural Networks
- 36. Scaling ● GraphSAGE https://arxiv.org/abs/1706.02216 ● ClusterGCN https://arxiv.org/abs/1905.07953 ● Graph-BERT https://arxiv.org/abs/2001.05140 https://github.com/jwzhanggy/Graph-Bert Additional reading: https://towardsdatascience.com/simple-scalable-graph-neural-networks-7eb04f366d07
- 37. Model Evaluation https://ogb.stanford.edu/ ● Datasets + Data Loaders (DGL / pyTorch Geometric) ● Node pred ● Edge pred ● Graph pred
- 38. Please feel free to ask questions ;)