PowerGraph

22.06.2015 DIMA – TU Berlin 1
Fachgebiet Datenbanksysteme und Informationsmanagement
Technische Universität Berlin
http://www.dima.tu-berlin.de/
Hot Topics in Information Management
PowerGraph: Distributed Graph-Parallel
Computation on Natural Graphs
Igor Shevchenko
Mentor: Sebastian Schelter

Agenda
1. Natural Graphs: Properties and Problems;
2. PowerGraph: Vertex Cut and Vertex Programs;
3. GAS Decomposition;
4. Vertex Cut Partitioning;
5. Delta Caching;
6. Applications and Evaluation;
Paper:
Gonzalez at al. PowerGraph: Distributed Graph-
Parallel Computation on Natural Graphs.

■ Natural graphs are graphs derived from real-world
or natural phenomena;
■ Graphs are big: billions of vertices and edges and
rich metadata;
Natural graphs have
Power-Law Degree Distribution
Natural Graphs

Power-Law Degree Distribution
(Andrei Broder et al. Graph structure in the web)

■ We want to analyze natural graphs;
■ Essential for Data Mining and Machine Learning;
Goal
Identify influential people and information;
Identify special nodes and communities;
Model complex data dependencies;
Target ads and products;
Find communities;
Flow scheduling;

■ Existing distributed graph computation systems
perform poorly on natural graphs (Gonzalez et al.
OSDI ’12);
■ The reason is presence of high degree vertices;
Problem
High Degree Vertices: Star-like motif

Possible problems with high degree vertices:
■ Limited single-machine resources;
■ Work imbalance;
■ Sequential computation;
■ Communication costs;
■ Graph partitioning;
Applicable to:
■ Hadoop; GraphLab; Pregel (Piccolo);
Problem Continued

■ High degree vertices can exceed the memory
capacity of a single machine;
■ Store edge meta-data and adjacency information;
Problem: Limited Single-Machine Resources

■ The power-law degree distribution can lead to
significant work imbalance and frequency barriers;
■ For ex. with synchronous execution (Pregel):
Problem: Work Imbalance

■ No parallelization of individual vertex-programs;
■ Edges are processed sequentially;
■ Locking does not scale well to high degree
vertices (for ex. in GraphLab);
Problem: Sequential Computation
Sequentially process edges Asynchronous execution requires heavy locking

■ Generate and send large amount of identical
messages (for ex. in Pregel);
■ This results in communication asymmetry;
Problem: Communication Costs

■ Natural graphs are difficult to partition;
■ Pregel and GraphLab use random (hashed)
partitioning on natural graphs thus maximizing
the network communication;
Problem: Graph Partitioning

■ Natural graphs are difficult to partition;
■ Pregel and GraphLab use random (hashed)
partitioning on natural graphs thus maximizing
the network communication;
Expected edges that are cut
Examples:
■ 10 machines:
■ 100 machines:
Problem: Graph Partitioning Continued
= number of machines
90% of edges cut;
99% of edges cut;

■ GraphLab and Pregel are not well suited for
computations on natural graphs;
Reasons:
■ Challenges of high-degree vertices;
■ Low quality partitioning;
Solution:
■ PowerGraph new abstraction;
In Summary

PowerGraph

Two approaches for partitioning the graph in a
distributed environment:
■ Edge Cut;
■ Vertex Cut;
Partition Techniques

■ Used by Pregel and GraphLab abstractions;
■ Evenly assign vertices to machines;
Edge Cut

■ Used by PowerGraph abstraction;
■ Evenly assign edged to machines;
Vertex Cut The strong point of the paper
4 edges 4 edges

Think like a Vertex
[Malewicz et al. SIGMOD’10]
User-defined Vertex-Program:
1. Runs on each vertex;
2. Interactions are constrained by graph structure;
Pregel and GraphLab also use this concept, where
parallelism is achieved by running multiple vertex
programs simultaneously;
Vertex Programs

■ Vertex cut distributes a single vertex-program
across several machines;
■ Allows to parallelize high-degree vertices;
GAS Decomposition The strong point of the paper

Generalize the vertex-program into three phases:
1. Gather
 Accumulate information about neighborhood;
2. Apply
 Apply accumulated value to center vertex;
3. Scatter
 Update adjacent edges and vertices;
GAS Decomposition
Gather, Apply and Scatter are user-defined functions;
The strong point of the paper

■ Executed on the edges in parallel;
■ Accumulate information about neighborhood;
Gather Phase

■ Executed on the central vertex;
■ Apply accumulated value to center vertex;
Apply Phase

■ Executed on the neighboring vertices in parallel;
■ Update adjacent edges and vertices;
Scatter Phase

■ Vertex-programs that are written using GAS
decomposition will automatically scale to several
machines; How does it work?
GAS Decomposition

GAS in a Distributed Environment

■ Case with 2 machines;
GAS in a Distributed Environment

■ Compute partial sums on each machine;
Gather Phase

■ Send partial sum to the master machine;
■ Master machine computes the total sum;
Gather Phase

■ Apply accumulated value to center vertex;
■ Replicate value to the mirrors;
Apply Phase

■ Update adjacent edges and vertices;
■ Initiate neighboring vertex-programs if necessary;
Scatter Phase

■ During the Gather Phase the partial results are
combined using commutative and associative
user-defined SUM operation;
■ Examples:
sum(a, b): return a + b
sum(a, b): return union(a, b)
sum(a, b): return min(a, b)
■ Also a requirement for Pregel combiners;
■ What if not commutative and associative?
SUM Operation

■ If not commutative and associative sum;
■ Send each edge data to the master machine;
■ Increases communication amount on Gather:
Gather Phase: no partial sums

Vertex Cut Partitioning

Three distributed approaches for Vertex Cut:
■ Random Edge Placement;
■ Coordinated Greedy Edge Placement;
■ Oblivious Greedy Edge Placemen;
Vertex Cut Partitioning
Minimize machines
spanned by each vertex
Minimize communication
and storage overhead
=

■ Randomly assign edges to machines;
Random Edge Placement

■ Edge data is uniquely assigned to one machine

■ Only 3 network communication channels;
■ Can predict network communication usage;
■ Significantly less communication comparing to the
Edge Cut graph placement;
■ Can improve upon random placement!
Communication Overhead

■ Place edges on machines which already has the
vertices in that edge;
Greedy Edge Placement

■ If several choices are possible, assign to the least
loaded machine;

■ Greedy Edge Placement is de-randomization;
■ Minimizes the number of machines spanned;
Coordinated Greedy Edge Placement:
■ Requires coordination to place each edge;
■ Maintains global distributed placement table;
■ Slower but produces higher quality cuts;
Oblivious Greedy Edge Placement:
■ Approx. greedy objective without coordination;
■ Faster but produces lower quality cuts;

■ Twitter Follower Graph: 41M vertices, 1.4B edges;
■ Oblivious Greedy Edge Placement balances cost
(replication factor) and construction time;
Vertex Cut Partitioning: Comparison

■ Greedy Edge Placement improves computation
performance;
Vertex Cut Partitioning: Comparison

Delta Caching
Execution Modes

■ Vertex-program can be triggered in response to a
change only in a few of its neighbors;
■ In response Gather Phase will accumulate
information about the all neighborhood;
Delta Caching The strong point of the paper

■ Accelerate the process by caching neighborhood
accumulators from previous gather phase;
Delta Caching The strong point of the paper

Delta Caching can speed up:
■ Gather Phase;
■ Scatter Phase;
Requires Abelian Group;
■ sum (+)
■ inverse (−)
Examples:
■ Page Rank – applicable;
■ Graph Coloring – not applicable;
Delta Caching
Commutative and associative

Supports three execution modes:
■ Synchronous: Bulk-Synchronous GAS Phases;
■ Asynchronous: Interleave GAS Phases;
■ Asynchronous Serializable: Prevent neighboring
vertices to run simultaneously;
Different tradeoffs:
■ Algorithm performance;
■ System performance;
■ Determinism;
Execution Modes

Evaluation

PowerGraph on the natural graphs shows:
■ Reduced network communication;
■ Reduced runtime;
■ Reduced storage;
On many examples
Evaluation
PageRank on the Twitter Follower Graph
(40M Users, 1.4 Billion Links)

■ Collaborative Filtering
 Alternating Least Squares
 Stochastic Gradient Descent
 SVD
 Non-negative MF
■ Statistical Inference
 Loopy Belief Propagation
 Max-Product Linear Programs
 Gibbs Sampling
Applicability
■ Graph Analytics
 PageRank
 Triangle Counting
 Shortest Path
 Graph Coloring
 K-core Decomposition
■ Computer Vision
 Image stitching
■ Language Modeling
 LDA

■ Vertex Cut;
■ GAS Decomposition;
■ Delta Caching;
■ Three modes of execution;
 Synchronous;
 Asynchronous;
 Asynchronous + Serializable;
Strong Points of the Paper

■ “In all cases the system is entirely symmetric with no single
coordinating instance or scheduler”;
How do they deal with Synchronous execution?
Evaluation mess:
■ Evaluated Synchronous execution using PageRank;
■ Evaluated Asynchronous execution using GraphColoring;
■ Evaluated Asynchronous+S execution using GraphColoring;
■ Compared PowerGraph with published results again using
PageRank, Triangle Counting but not GraphColoring;
■ Oblivious Greedy Edge Placement is poorly explained;
Weak Points of the Paper

■ Joseph E. Gonzalez, Yucheng Low, Haijie Gu, Danny
Bickson, Carlos Guestrin. PowerGraph: Distributed Graph-
Parallel Computation on Natural Graphs. 10th USENIX
Symposium on Operating Systems Design and
Implementation (OSDI 2012);
■ Malewicz, G., Austern, M. H., Bik, A. J., Dehnert, J., Horn, I.,
Leiser, N., and Czajkowski, G. Pregel: a system for large-
scale graph processing. In SIGMOD (2010).
■ Low, Y., Gonzalez, J., Kyrola, A., Bickson, D., Guestrin, C.,
and Hellerstein, J. M. Distributed GraphLab: A Framework
for Machine Learning and Data Mining in the Cloud. in
PVLDB (2012).
■ http://graphlab.org
References

Questions?
1. Natural Graphs: Properties and Problems;
2. PowerGraph: Vertex Cut and Vertex Programs;
3. GAS Decomposition;
4. Vertex Cut Partitioning;
5. Delta Caching;
6. Applications and Evaluation;
Paper:
Gonzalez at al. PowerGraph: Distributed Graph-
Parallel Computation on Natural Graphs.

PowerGraph

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to PowerGraph (20)

PowerGraph