Understanding Large Social Networks | IRE Major Project | Team 57 | LINE
- 1. Understanding Large Social Networks Information Retrieval and Extraction (Major Project) Team Members (Team No.: 57) Madan Jhawar 201202018 Raj Patel 201301048 Kishan Vadaliya 201505621 Mentor Ganesh Jawahar
- 2. Problem Statement To build a model that can capture the network information of a node in an efficient and scalable manner by representing every node into low- dimensional vector spaces.
- 3. Motivation ● Representing network nodes into low-dimensional vector spaces helps us in many tasks such as visualization, node classification, and link prediction, etc., using standard machine learning algorithms on top of that. ● The size of most real-world information networks such as YouTube, Flickr, Facebook, etc., ranges from hundreds of nodes to millions and billions of nodes. Scalable solutions to understand such networks are very useful in the industry. ● Deep Learning has worked wonders in Language Modelling and has provided commendable results in node representation tasks as well.
- 4. Dataset used ● Blogcatalog dataset is used for this project. ● It contains 10312 nodes, 333983 links. ● These nodes and links represent users and their friendship respectively. ● Dataset - http://socialcomputing.asu.edu/datasets/BlogCatalog3
- 5. Baseline Paper LINE: LARGE-SCALE INFORMATION NETWORK EMBEDDING Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, Qiaozhu Mei
- 6. LINE: First-Order Proximity ● First-order proximity refers to the local pairwise proximity between the vertices in the network (i.e Only the direct neighbours). ● To model the first-order proximity between each undirected edge (i, j), we define ‘joint probability’ between vertex vi and vj , where the empirical probability is given by: ● Minimize the KL-divergence of two probability distributions, given by
- 7. LINE: Second-Order Proximity ● The second-order proximity assumes that vertices sharing many connections to other vertices are similar to each other. ● In this model - Vertex plays two roles: the vertex itself( ) and a specific “context” ( ) of other vertices. ● For each directed edge (i, j), we first define the probability of “context” vj generated by vertex vi as: , - where the empirical probability is given by: ● We take KL-divergence as the distance function:
- 8. LINE: Negative Sampling (Optimization) ● If we carefully analyze the denominator, it is quite expensive operation as it considers dot-product with every other node, where dot-product itself takes linear (depends on size of the vector) time. ● To optimize we use Negative Sampling, which samples multiple negative edges according to some noisy distribution for each edge (i, j)
- 9. Novelty ● The authors train the neural networks for the First-Order Proximity and the Second-Order Proximity independently, and then concatenate the embeddings learned from both the models for a node to get the final embeddings of a node. ● In our project, we use the same lookup table for the node representation in both the neural networks, i.e., the node embeddings learned from the First-Order Proximity are used in the neural network for the Second-Order Proximity, instead of initializing them again to random values. ● The embeddings learned from the Second-Order Proximity deep-net are, then, taken as the final embeddings of a node.
- 10. Languages used ● The complete code is written in Lua language. ● For building Neural Networks, Torch, a framework written in Lua, is used. ● We use the scikit-learn library in Python to test the accuracy of our model.
- 11. Results ● Parameters: Feature Value Embedding Size 10 Number of Epochs 30 Learning Rate 0.035 Number of Negative Samples 10
- 12. Train% 60% 70% 80% Micro-F1 39.34% 39.11% 41.62% Macro-F1 23.82% 24.13% 25.36% Results Contd.. Metric ● F-Scores:
- 13. Week 1 Getting started with NN and Deep Learning Week 2 Formalizing the problem and getting acquainted with the baseline paper Week 3 Implementing first-order proximity in Torch. Week 4 Implementing second-order proximity in Torch. Week 5 Combining first-order and second-order along with extending the work Project Timeline
- 14. Challenges Faced ● Understanding the mathematics of the baseline paper. ● Referring to other research articles related to the baseline paper. ● Getting around with Lua and Torch. ● Not everything could be printed and debugged while coding neural nets in Lua. ● The dataset is very large, so we had to make several adjustments with the data structures used in our code to reduce the time complexity.
- 15. References ● http://www.www2015.it/documents/proceedings/proceedings/p1067.pdf ● http://arxiv.org/pdf/1403.6652.pdf ● http://cs224d.stanford.edu/lecture_notes/LectureNotes1.pdf ● https://github.com/torch/nn ● https://github.com/gjwhr/dl4nlp-made-easy/blob/master/word2vec/cbow.lua ● https://github.com/gjwhr/dl4nlp-made-easy/blob/master/word2vec/skipgram.lua
- 17. Thank you!