NS-CUK Joint Journal Club: V.T.Hoang, Review on "NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs", ICLR 2023
- 1. 2023. 03. 16 Van Thuy Hoang Network Science Lab Dept. of Artificial Intelligence The Catholic University of Korea E-mail: hoangvanthuy90@gmail.com
- 2. 2 Program Flow Ecological Network Biological Network Social Network Chemical Network Web Graph Graphs are everywhere.
- 3. 3 • The goal: • Mapping individual nodes to vector points in latent space. Graph embedding learning
- 4. 4 • GNN-based approaches provide faster and practical training and state-of-the-art results on benchmark datasets for downstream tasks such as node classification GNN-based models
- 5. 5 GNN aggregation Aggregation from neighbourhoods
- 6. 6 Problems Most of the existing GNN architectures have two fundamental weaknesses which restrict their learning ability on general graph-structured data: Over-smoothing problems GNNs fail to be deep enough: Trade-off between features and structure Noise from neighbours GNNs seem to be tailor-made to work on homophilic (associative) graphs
- 7. 7 Problems The existing graph transformers: Treat the nodes as independent tokens Construct a single sequence of all the node tokens to train the model
- 8. 8 Contributions Training such models on large graphs will cost a huge GPU resources NAGphormer: Hop2Token Attention-based readout function
- 9. 9 Model framework of NAGphormer XXX
- 10. 10 Generate sequences of all nodes in graphs XXX
- 11. 11 Details Transformer layer: Readout function:
- 12. 12 IMPLEMENTATION DETAILS Structural encoding Besides the attribute information of nodes, the structural information of nodes is also a crucial feature for graph mining tasks the eigenvectors of Laplacian matrix of the graph for capturing the structural information of nodes
- 13. 13 EXPERIMENTS Graph datasets Baselines: GNNs Graph transformers
- 16. 16 Ablation study Structural encoding
- 17. 17 Ablation study Different readout functions.
- 18. 18 Pros & Cons Pros What Hop2Token can do: Capture global information (up to K-hop) Solve isomophic subgraphs Attention layer: learning more informative node representations from the multi-hop neighborhoods Cons: Expressive: <= 1-d Weisfeiler lehman test Fails to capture graph structure Noise from neighbourhood