Joey gonzalez, graph lab, m lconf 2013
3 likes3,161 views
The document discusses machine learning techniques for graphs and graph-parallel computing. It describes how graphs can model real-world data with entities as vertices and relationships as edges. Common machine learning tasks on graphs include identifying influential entities, finding communities, modeling dependencies, and predicting user behavior. The document introduces the concept of graph-parallel programming models that allow algorithms to be expressed by having each vertex perform computations based on its local neighborhood. It presents examples of graph algorithms like PageRank, product recommendations, and identifying leaders that can be implemented in a graph-parallel manner. Finally, it discusses challenges of analyzing large real-world graphs and how systems like GraphLab address these challenges through techniques like vertex-cuts and asynchronous execution.
![Recommending Products
≈
Movies
f(j)
f(i)
Movies
Iterate:
f [i] = arg min
w2Rd
X
j2Nbrs(i)
rij
f(1)
User Factors (U)
Users
Netflix
x
f(2)
T
w f [j]
r13
r14
r24
r25
2
f(3)
f(4)
f(5)
Movie Factors (M)
Users
Low-Rank Matrix Factorization:
+ ||w||2
2
8](https://image.slidesharecdn.com/joeygonzalezgraphlabmlconf2013-131119180112-phpapp01/85/Joey-gonzalez-graph-lab-m-lconf-2013-8-320.jpg)
![Identifying Leaders
R[i] = 0.15 +
X
wji R[j]
j2Nbrs(i)
Rank of
user i
Weighted sum of
neighbors’ ranks
Everyone starts with equal ranks
Update ranks in parallel
Iterate until convergence
10](https://image.slidesharecdn.com/joeygonzalezgraphlabmlconf2013-131119180112-phpapp01/85/Joey-gonzalez-graph-lab-m-lconf-2013-10-320.jpg)
![How should we program"
graph-parallel algorithms?
“Think like a Vertex.”
- Pregel [SIGMOD’10]
15](https://image.slidesharecdn.com/joeygonzalezgraphlabmlconf2013-131119180112-phpapp01/85/Joey-gonzalez-graph-lab-m-lconf-2013-15-320.jpg)
![The Graph-Parallel Abstraction
A user-defined Vertex-Program runs on each vertex
Graph constrains interaction along edges
Using messages (e.g. Pregel [PODC’09, SIGMOD’10])
Through shared state (e.g., GraphLab [UAI’10, VLDB’12])
Parallelism: run multiple vertex programs simultaneously
16](https://image.slidesharecdn.com/joeygonzalezgraphlabmlconf2013-131119180112-phpapp01/85/Joey-gonzalez-graph-lab-m-lconf-2013-16-320.jpg)
![The GraphLab Vertex Program
Vertex Programs directly access adjacent vertices and edges
GraphLab_PageRank(i)
//
Compute
sum
over
neighbors
total
=
0
foreach
(j
in
neighbors(i)):
total
=
total
+
R[j]
*
wji
//
Update
the
PageRank
R[i]
=
0.15
+
total
//
Trigger
neighbors
to
run
again
if
R[i]
not
converged
then
signal
nbrsOf(i)
to
be
recomputed
R[4]
*
w41
4
+
1
+
3
Signaled vertices are recomputed eventually.
2
17](https://image.slidesharecdn.com/joeygonzalezgraphlabmlconf2013-131119180112-phpapp01/85/Joey-gonzalez-graph-lab-m-lconf-2013-17-320.jpg)
![A Common Pattern in
Vertex Programs
GraphLab_PageRank(i)
//
Compute
sum
over
neighbors
total
=
0
Gather
Informa1on
foreach(
j
in
neighbors(i)):
About
Neighborhood
total
=
total
+
R[j]
*
wji
//
Update
the
PageRank
Update
Vertex
R[i]
=
total
//
Trigger
neighbors
to
run
again
priority
=
|R[i]
–
oldR[i]|
Signal
Neighbors
&
if
R[i]
not
converged
then
Modify
Edge
Data
signal
neighbors(i)
with
priority
27](https://image.slidesharecdn.com/joeygonzalezgraphlabmlconf2013-131119180112-phpapp01/85/Joey-gonzalez-graph-lab-m-lconf-2013-27-320.jpg)
![Minimizing Communication in
PowerGraph
Y
Communication is linear in "
the number of machines "
each vertex spans
A vertex-cut minimizes "
machines each vertex spans
Percolation theory suggests that power law
graphs have good vertex cuts. [Albert et al. 2000]
29](https://image.slidesharecdn.com/joeygonzalezgraphlabmlconf2013-131119180112-phpapp01/85/Joey-gonzalez-graph-lab-m-lconf-2013-29-320.jpg)
![PageRank on Twitter Follower Graph
Natural Graph with 40M Users, 1.4 Billion Links
Run1me
Per
Itera1on
0
50
100
150
200
Hadoop
GraphLab
Twister
Piccolo
Order of magnitude
by exploiting
properties of Natural
Graphs
PowerGraph
Hadoop results from [Kang et al. '11]
Twister (in-memory MapReduce) [Ekanayake et al. ‘10]
31](https://image.slidesharecdn.com/joeygonzalezgraphlabmlconf2013-131119180112-phpapp01/85/Joey-gonzalez-graph-lab-m-lconf-2013-31-320.jpg)
![Triangle Counting on Twitter
40M Users, 1.4 Billion Links
Counted: 34.8 Billion Triangles
Hadoop
[WWW’11]
1536 Machines
423 Minutes
64 Machines
15 Seconds
1000 x Faster
34
S.
Suri
and
S.
Vassilvitskii,
“CounMng
triangles
and
the
curse
of
the
last
reducer,”
WWW’11](https://image.slidesharecdn.com/joeygonzalezgraphlabmlconf2013-131119180112-phpapp01/85/Joey-gonzalez-graph-lab-m-lconf-2013-34-320.jpg)
Joey gonzalez, graph lab, m lconf 2013
- 1. Machine Learning on Graphs Joseph Gonzalez Co-Founder, GraphLab Inc. joseph@graphlab.com Postdoc, UC Berkeley AMPLab jegonzal@eecs.berkeley.edu
- 2. Big Data Graphs More Signal More Noise 2!
- 3. Social Media Science Advertising Web Graphs encode relationships between: People Products Ideas Facts Interests Big: billions of vertices and edges & rich metadata Facebook (10/2012): 1B users, 144B friendships Twi>er (2011): 15B follower edges 3
- 4. Graphs are Essential to " Data Mining and Machine Learning Identify influential people and information Find communities Understand people’s shared interests Model complex data dependencies
- 5. Predicting User Behavior ? ? ? Liberal ? ? ? Conservative ? ? ? Post Post ? ? Post Post Post ? Post ? Post Post ? ? ? Post ? Post Post Post ? Conditional Random Field! ? ? ? ? ? ? Belief Propagation! Post ? ? Post Post Post ? ? ? ? 5
- 6. Finding Communities Count triangles passing through each vertex: " 2 3 1 4 Measures “cohesiveness” of local community Fewer Triangles Weaker Community More Triangles Stronger Community
- 8. Recommending Products ≈ Movies f(j) f(i) Movies Iterate: f [i] = arg min w2Rd X j2Nbrs(i) rij f(1) User Factors (U) Users Netflix x f(2) T w f [j] r13 r14 r24 r25 2 f(3) f(4) f(5) Movie Factors (M) Users Low-Rank Matrix Factorization: + ||w||2 2 8
- 10. Identifying Leaders R[i] = 0.15 + X wji R[j] j2Nbrs(i) Rank of user i Weighted sum of neighbors’ ranks Everyone starts with equal ranks Update ranks in parallel Iterate until convergence 10
- 11. Graph-Parallel Algorithms Model / Alg. State Computation depends only on the neighbors 11
- 12. Many More Graph Algorithms • Collaborative Filtering! – – – – • Graph Analytics! Alternating Least Squares! Stochastic Gradient Descent! Tensor Factorization! SVD! • Structured Prediction! – Loopy Belief Propagation! – Max-Product Linear Programs! – Gibbs Sampling! • Semi-supervised ML! – Graph SSL ! – CoEM! – – – – – – PageRank! Shortest Path! Triangle-Counting! Graph Coloring! K-core Decomposition! Personalized PageRank! • Classification! – Neural Networks! – Lasso! …! 12
- 13. How should we program" graph-parallel algorithms? 13
- 14. Structure of Computation Data-Parallel Graph-Parallel Dependency Graph Table Row 6. Before Row Row Result 7. After Row 14 8. After
- 15. How should we program" graph-parallel algorithms? “Think like a Vertex.” - Pregel [SIGMOD’10] 15
- 16. The Graph-Parallel Abstraction A user-defined Vertex-Program runs on each vertex Graph constrains interaction along edges Using messages (e.g. Pregel [PODC’09, SIGMOD’10]) Through shared state (e.g., GraphLab [UAI’10, VLDB’12]) Parallelism: run multiple vertex programs simultaneously 16
- 17. The GraphLab Vertex Program Vertex Programs directly access adjacent vertices and edges GraphLab_PageRank(i) // Compute sum over neighbors total = 0 foreach (j in neighbors(i)): total = total + R[j] * wji // Update the PageRank R[i] = 0.15 + total // Trigger neighbors to run again if R[i] not converged then signal nbrsOf(i) to be recomputed R[4] * w41 4 + 1 + 3 Signaled vertices are recomputed eventually. 2 17
- 18. Num-‐Ver1ces Be>er Convergence of Dynamic PageRank 100000000 51% updated only once! 1000000 10000 100 1 0 10 20 30 40 Number of Updates 50 60 70 18
- 19. Adaptive Belief Propagation Challenge = Boundaries Many Updates Splash Noisy “Sunset” Image Few Updates Cumulative Vertex Updates Algorithm identifies and focuses on hidden sequential structure Graphical Model
- 20. 6. Before Graph-‐parallel Abstrac(ons BeDer for Machine Learning Messaging i Synchronous 7. After 8. After Shared State i Dynamic Asynchronous 20
- 21. Natural Graphs Graphs derived from natural phenomena 21
- 22. Properties of Natural Graphs Regular Mesh Natural Graph Power-Law Degree Distribution 22
- 23. Power-Law Degree Distribution “Star Like” Motif President Obama Followers 23
- 24. Challenges of High-‐Degree VerMces SequenMally process edges Touches a large fracMon of graph CPU 1 CPU 2 Provably Difficult to ParMMon 24
- 25. ment. While fast and easy to implement, placement cuts most of the edges: Random ParMMoning em 5.1. If vertices random (hashed) assigne are randomly • GraphLab resorts to parMMoning on natural graphs nes then the expected fraction of edges cut |Edges Cut| E =1 |E| 1 p 10 Machines ! 90% of edges cut example if just two machines are used, hal 100 Machines ! 99% of edges cut! Machine 1 Machine 2 es will be cut requiring order |E| /2 commun 25
- 26. Program For This Run on This Machine 1 Machine 2 • Split High-‐Degree verMces • New Abstrac1on ! Equivalence on Split Ver(ces 26
- 27. A Common Pattern in Vertex Programs GraphLab_PageRank(i) // Compute sum over neighbors total = 0 Gather Informa1on foreach( j in neighbors(i)): About Neighborhood total = total + R[j] * wji // Update the PageRank Update Vertex R[i] = total // Trigger neighbors to run again priority = |R[i] – oldR[i]| Signal Neighbors & if R[i] not converged then Modify Edge Data signal neighbors(i) with priority 27
- 28. GAS Decomposition Machine 1 Machine 2 Master Gather Apply Sca>er Y’ Y’ Y’ Y’ Σ1 Σ Σ2 + + + Mirror Y Σ3 Σ4 Mirror Machine 3 Mirror Machine 4 28
- 29. Minimizing Communication in PowerGraph Y Communication is linear in " the number of machines " each vertex spans A vertex-cut minimizes " machines each vertex spans Percolation theory suggests that power law graphs have good vertex cuts. [Albert et al. 2000] 29
- 30. Machine Learning and Data-Mining Toolkits Graph AnalyMcs Graphical Models Computer Vision Clustering Topic Modeling CollaboraMve Filtering GraphLab2 System MPI/TCP-‐IP PThreads HDFS EC2 HPC Nodes http://graphlab.org Apache 2 License
- 31. PageRank on Twitter Follower Graph Natural Graph with 40M Users, 1.4 Billion Links Run1me Per Itera1on 0 50 100 150 200 Hadoop GraphLab Twister Piccolo Order of magnitude by exploiting properties of Natural Graphs PowerGraph Hadoop results from [Kang et al. '11] Twister (in-memory MapReduce) [Ekanayake et al. ‘10] 31
- 32. GraphLab2 is Scalable Yahoo Altavista Web Graph (2002): One of the largest publicly available web graphs 1.4 Billion Webpages, 6.6 Billion Links 7 Seconds per Iter. 1B links Nodes processed per second 64 HPC 1024 Cores (2048 30 lines of user code HT) 32
- 33. Topic Modeling English language Wikipedia – 2.6M Documents, 8.3M Words, 500M Tokens – Computationally intensive algorithm Million Tokens Per Second 0 Smola et al. PowerGraph 20 40 60 80 100 120 140 160 100 Yahoo! Machines Specifically engineered for this task 64 cc2.8xlarge EC2 Nodes 200 lines of code & 4 human hours 33
- 34. Triangle Counting on Twitter 40M Users, 1.4 Billion Links Counted: 34.8 Billion Triangles Hadoop [WWW’11] 1536 Machines 423 Minutes 64 Machines 15 Seconds 1000 x Faster 34 S. Suri and S. Vassilvitskii, “CounMng triangles and the curse of the last reducer,” WWW’11
- 35. 7. After 8. After By exploiting common patterns in graph data and computation: New ways to represent real-world graphs New ways execute graph algorithms Machine 1 Machine 2 Orders of magnitude improvements over existing systems
- 37. Exciting Time to Work in ML J Unique opportunities to change the world!! With ML, I will cure cancer!!! With ML I will find true love. Why won’t ML read my mind??? L Building scalable learning system requires experts …
- 38. But… Even basics of scalable ML can be challenging ML key to any new service we want to build 6 months from prototype to producMon State-‐of-‐art ML algorithms trapped in research papers Goal of GraphLab 3: Make large-scale machine learning accessible to all! J
- 39. Adding a Python Layer Python API Graph AnalyMcs Graphical Models Computer Vision Clustering Topic Modeling CollaboraMve Filtering GraphLab2 System MPI/TCP-‐IP PThreads EC2 HPC Nodes HDFS
- 40. Learning ML with GraphLab Notebook https://beta.graphlab.com/examples!
- 41. Prototype to Production with Python GraphLab: Easily install prototype locally Deploy to the cluster in one step
- 42. Learn: GraphLab Notebook Prototype: pip install graphlab è local prototyping Production: Same code scales to EC2 cluster
- 43. GraphLab Toolkits Highly scalable, state-of-the-art machine learning straight from python Graph Analytics Graphical Models Computer Vision Clustering Topic Modeling Collaborative Filtering
- 44. Machine Learning on Graphs partners@graphlab.com NIPS Workshop on Big Learning: biglearn.org Lake Tahoe, December 9th Joseph Gonzalez Co-Founder, GraphLab Inc. joseph@graphlab.com