NS-CUK Seminar: H.B.Kim, Review on "Inductive Representation Learning on Large Graphs", NIPS2017
- 1. Ho-Beom Kim Network Science Lab Dept. of Mathematics The Catholic University of Korea E-mail: hobeom2001@catholic.ac.kr 2023 / 08 / 07 HAMILTON, Will; YING, Zhitao; LESKOVEC Advances in neural information processing systmens
- 2. 2 Introduction Problem Statements 1. The basic idea behind node embedding approaches is to use dimensionality reduction techniques to distill the high-dimensional information about a node’s graph neighborhood into a dense vector embedding. 2. previous works have focused on embedding nodes from a single fixed graph, and many real-world applications require embeddings to be quickly generated for unseen nodes, or entirely new (sub)graphs. 3. The inductive node embedding problem is especially difficult, compared to the transductive setting, because generalizing to unseen nodes requires “aligning” newly observed subgraphs to the node embeddings that the algorithm has already optimized on. 4. Most existing approaches to generating node embeddings are inherently transductive.
- 3. 3 Introduction Contributions 1. They propose a general framework, called GraphSAGE, for inductive node embedding 2. They leverage node features in order to learn an embedding function that generalizes to unseen nodes 3. Their algorithm can be applied to graphs without node features 4. Across domains, their supervised approach improves classification F1-scores by an average of 51% compared to using node features alone and GraphSAGE consistently outperforms a strong
- 4. 4 Methodology Visual illustration of the GraphSAGE sample and aggregate approach
- 5. 5 Methodology GraphSAGE embedding generation algorithm
- 6. 6 Methodology GraphSAGE embedding generation algorithm
- 7. 7 Methodology Neighborhood definition • They uniformly sample a fixed-size set of neighbors, instead of using full neighborhood sets • They define N (v) as a fixed-size, uniform draw from the set {u ∈ V : (u, v) ∈ E}, and they draw different uniform samples at each iteration, k • Without this sampling the memory and expected runtime of a single batch is unpredictable and in the worst case O(|V|). • the per-batch space and time complexity for GraphSAGE is fixed at O( QK i=1 Si), where Si , i ∈ {1, ..., K} and K are user-specified constants
- 8. 8 Methodology Learning the parameters of GraphSAGE
- 9. 9 Methodology Aggregator Architectures • Mean aagregator • LSTM aggregator • Pooling aggregator
- 10. 10 Experiments Datasets • Citation data • Reddit data
- 11. 11 Experiments Prediction results for the three datasets
- 12. 12 Experiments Prediction results for the three datasets
- 13. 13 Conclusion Conclusion • They introduced a novel approach that allows embeddings to be efficiently generated for unseen nodes. • GraphSAGE consistently outperforms state-of-the-art baselines, effectively trades off performance and runtime by sampling node neighborhoods, and their theoretical analysis provides insight into how their approach can learn about local graph structures. • A number of extensions and potential improvements are possible, such as extending GraphSAGE to incorporate directed or multi-modal graphs. • A particularly interesting direction for future work is exploring non-uniform neighborhood sampling functions, and perhaps even learning these functions as part of the GraphSAGE optimization.
Editor's Notes
- #5: Let's say the red node in the middle of the picture above is the newly added node. We need to find the embedding of this red node. First, a certain number of neighborhood nodes are sampled based on the distance (k). Then, through the aggregation function learned through graphsage, the embedding of the red node is calculated from the features of the surrounding nodes. The embedding of the inferred new node is used for the downstream task.
- #6: 현재 표현 h_(k−1)v와 집계된 이웃 벡터 h_(k−1)N(v)를 연결 To extend Algorithm 1 to the minibatch setting, given a set of input nodes, we first forward sample the required neighborhood sets (up to depth K) and then we run the inner loop (line 3 in Algorithm 1), but instead of iterating over all nodes, we compute only the representations that are necessary to satisfy the recursion at each depth (Appendix A contains complete minibatch pseudocode).
- #7: In the process of learning GraphSAGE, operations need to be performed in batch units. Therefore, in this paper, the following batch sampling algorithm (lines 1 to 7) is utilized.
- #9: v is a node that co-occurs near u on fixed-length random walk, σ is the sigmoid function, Pn is a negative sampling distribution, and Q defines the number of negative sample the representations zu that we feed into this loss function are generated from the features contained within a node’s local neighborhood, rather than training a unique embedding for each no
- #10: A function that takes the average of the elements of a vector. In the case of LSTM, it has an advantage in expressiveness, but it is not permutation invariant because it is not inherently symmetric.Therefore, in this paper, LSTM works well even for unordered vector sets by applying LSTM to the random permutation of the node's neighbors. Pooling Aggregators are both symmetric and learnable. Each neighbor vector is independently fed into a fully-connected neural network. Afterwards, Elementwise max-pooling operation is applied to the neighbor set to integrate the information. In theory, multiple layers can be stacked before max-pooling, but in this paper, only one layer was simply used, and this method is effective in terms of efficiency. show a better appearance inBy applying a max-pooling operation to each computed feature, the model effectively captures different aspects of the set of neighbors. Of course, any symmetric vector function can be used instead of the max operator.In this paper, no significant difference was found between max-pooling and mean-pooling, and in subsequent papers, the process was unified by applying max-pooling.
- #11: undirected graph data node : papers / edge : citation between papers task : classifying category of the paper(node classification) node features : node degree, paper abstract(GloVe embedding) This data is an evolving graph in which new nodes are added over time. In the experiment, after learning graphsage with graphs from 2000 to 2004, the graphs from 2005 are used for evaluation. undirected graph data node : reddit post / edge : if same user comments on both task : classifying community of the post(node classification) node features : embedding of post title, post's comment, score of post, number of comments on the post This dataset is a dataset made up of reddit posts. If there are two posts and the same user comments on both posts at the same time, the two posts (nodes) are connected. This graph consists of 232,965 posts (nodes) during September 2014, of which the graphs of the first 20 days were used for graphsage learning and the graphs of the remaining 10 days were used for evaluation.
- #12: In this paper, the performance of GraphSAGE was evaluated on a total of three benchmark tasks. (1) Classifying academic papers into different categories using the Web of Science citation dataset (2) Identifying the community to which posts on Reddit belong (3) Distinguishing protein functions in various biological protein-protein interaction graphs In the case of this chapter, I hope you will directly refer to the thesis, and I will organize only a few points. First of all, a total of four methodologies were used as a comparison group for GraphSAGE. There are four types: completely random, logistic regression using only raw features without considering graph structure, DeepWalk, and finally DeepWalk + raw features. GraphSAGE also experimented with a total of four styles. There are four types: GCN structure, mean aggregator structure, LSTM aggregator structure, and pooling aggregator structure. Except for DeepWalk, which used vanilla Gradient Descent Optimizer, Adam Opimizer was applied. Also, for fair comparison, all models were operated in the same mini-batch environment.
- #13: Aggregators based on LSTM and Pooling showed the best performance. Setting K = 2 showed a good performance in terms of efficiency, and sub-sampling of neighboring entities is a necessary step because although it makes the variance large, it greatly shortens the time.