Demystifying Distributed Graph Processing
8 likes3,054 views
The document discusses the challenges and misconceptions of distributed graph processing, explaining why traditional single-node data handling is often inadequate for large datasets. It outlines diverse applications and historical developments in distributed graph processing methods, including Pregel and PowerGraph, and highlights the Apache Flink graph API, which integrates these concepts for efficient data processing. Key topics include graph traversal, ego-network analysis, and pattern matching, emphasizing the need for powerful tools in managing big data graphs.
Demystifying Distributed Graph Processing
- 4. MY GRAPH IS SO BIG, IT DOESN’T FIT IN A SINGLE MACHINE Big Data Ninja MISCONCEPTION #1
- 6. YOUR INPUT DATASET SIZE IS _OFTEN_ IRRELEVANT
- 7. INTERMEDIATE DATA: THE OFTEN DISREGARDED EVIL ▸ Naive Who(m) to Follow: ▸ compute a friends-of-friends list per user ▸ exclude existing friends ▸ rank by common connections
- 8. DISTRIBUTED PROCESSING IS ALWAYS FASTER THAN SINGLE-NODE Data Science Rockstar MISCONCEPTION #2
- 10. GRAPHS DON’T APPEAR OUT OF THIN AIR Expectation…
- 11. GRAPHS DON’T APPEAR OUT OF THIN AIR Reality!
- 12. HOW DO WE EXPRESS A DISTRIBUTED GRAPH ANALYSIS TASK?
- 13. GRAPH APPLICATIONS ARE DIVERSE ▸ Iterative value propagation ▸ PageRank, Connected Components, Label Propagation ▸ Traversals and path exploration ▸ Shortest paths, centrality measures ▸ Ego-network analysis ▸ Personalized recommendations ▸ Pattern mining ▸ Finding frequent subgraphs
- 14. RECENT DISTRIBUTED GRAPH PROCESSING HISTORY 2004 MapReduce Pegasus 2009 Pregel 2010 Signal-Collect PowerGraph 2012
- 15. RECENT DISTRIBUTED GRAPH PROCESSING HISTORY 2004 MapReduce Pegasus 2009 Pregel 2010 Signal-Collect PowerGraph 2012 Iterative value propagation
- 16. PREGEL: THINK LIKE A VERTEX 1 5 4 3 2 1 3, 4 2 1, 4 5 3 ...
- 17. PREGEL: SUPERSTEPS (Vi+1, outbox) <— compute(Vi, inbox) 1 3, 4 2 1, 4 5 3 ... 1 3, 4 2 1, 4 5 3 ... Superstep i Superstep i+1
- 18. PREGEL EXAMPLE: PAGERANK void compute(messages): sum = 0.0 for (m <- messages) do sum = sum + m end for setValue(0.15/numVertices() + 0.85*sum) for (edge <- getOutEdges()) do sendMessageTo( edge.target(), getValue()/numEdges) end for sum up received messages update vertex rank distribute rank to neighbors
- 19. SIGNAL-COLLECT outbox <— signal(Vi) 1 3, 4 2 1, 4 5 3 ... 1 3, 4 2 1, 4 5 3 ... Superstep i Vi+1 <— collect(inbox) 1 3, 4 2 1, 4 5 3 ... Signal Collect Superstep i+1
- 20. SIGNAL-COLLECT EXAMPLE: PAGERANK void signal(): for (edge <- getOutEdges()) do sendMessageTo( edge.target(), getValue()/numEdges) end for void collect(messages): sum = 0.0 for (m <- messages) do sum = sum + m end for setValue(0.15/numVertices() + 0.85*sum) distribute rank to neighbors sum up received messages update vertex rank
- 21. GATHER-SUM-APPLY (POWERGRAPH) 1 ... ... Gather Sum 1 2 5 ... Apply 3 1 5 5 3 1 ... Gather 3 1 5 5 3 Superstep i Superstep i+1
- 22. GSA EXAMPLE: PAGERANK double gather(source, edge, target): return target.value() / target.numEdges() double sum(rank1, rank2): return rank1 + rank2 double apply(sum, currentRank): return 0.15 + 0.85*sum compute partial rank combine partial ranks update rank
- 23. RECENT DISTRIBUTED GRAPH PROCESSING HISTORY 2004 MapReduce Pegasus 2009 Pregel 2010 Signal-Collect PowerGraph 2012 Iterative value propagation Giraph++ 2013
- 24. RECENT DISTRIBUTED GRAPH PROCESSING HISTORY 2004 MapReduce Pegasus 2009 Pregel 2010 Signal-Collect PowerGraph 2012 Iterative value propagation Giraph++ 2013 Graph Traversals
- 25. THINK LIKE A (SUB)GRAPH 1 5 4 3 2 1 5 4 3 2 - compute() on the entire partition - Information flows freely inside each partition - Network communication between partitions, not vertices
- 26. RECENT DISTRIBUTED GRAPH PROCESSING HISTORY 2004 MapReduce Pegasus 2009 Pregel 2010 Signal-Collect PowerGraph 2012 Iterative value propagation Giraph++ 2013 Graph Traversals NScale 2014
- 27. RECENT DISTRIBUTED GRAPH PROCESSING HISTORY 2004 MapReduce Pegasus 2009 Pregel 2010 Signal-Collect PowerGraph 2012 Iterative value propagation Giraph++ 2013 Graph Traversals NScale 2014 Ego-network analysis Arabesque 2015 Tinkerpop
- 28. RECENT DISTRIBUTED GRAPH PROCESSING HISTORY 2004 MapReduce Pegasus 2009 Pregel 2010 Signal-Collect PowerGraph 2012 Iterative value propagation Giraph++ 2013 Graph Traversals NScale 2014 Ego-network analysis Arabesque 2015 Pattern Matching Tinkerpop
- 30. CAN WE HAVE IT ALL? ▸ Data pipeline integration: built on top of an efficient distributed processing engine ▸ Graph ETL: high-level API with abstractions and methods to transform graphs ▸ Familiar programming model: support popular programming abstractions
- 31. HELLO, GELLY! THE APACHE FLINK GRAPH API ▸ Java and Scala APIs: seamlessly integrate with Flink’s DataSet API ▸ Transformations, library of common algorithms val graph = Graph.fromDataSet(edges, env) val ranks = graph.run(new PageRank(0.85, 20)) ▸ Iteration abstractions Pregel Signal-Collect Gather-Sum-Apply Partition-Centric*
- 32. POSIX Java/Scala Collections POSIX ‣efficient streaming runtime ‣native iteration operators ‣well-integrated WHY FLINK?
- 33. FEELING GELLY? ▸ Paper References http://www.citeulike.org/user/vasiakalavri/tag/dotscale ▸ Apache Flink: http://flink.apache.org/ ▸ Gelly documentation: http://ci.apache.org/projects/flink/flink-docs-master/apis/batch/libs/gelly.html ▸ Gelly-Stream: https://github.com/vasia/gelly-streaming