Dynamic Draph / Iterative Computation on Apache Giraph
6 likes1,777 views
Dynamic graph / iterative computation on Apache Giraph. Apache Giraph is an iterative graph processing system inspired by Pregel that runs on Hadoop. It allows modeling problems as a series of messages passed between graph vertices. Giraph has been used for applications like page rank, affinity propagation, and mutual friends calculation. It provides faster computation than Hive and scales to trillions of edges. Future work includes automatic checkpointing and investigating alternative computing models.
Dynamic Draph / Iterative Computation on Apache Giraph
- 1. Dynamic graph / iterative computation on Apache Giraph 6/3/2014 Avery Ching Hadoop Summit
- 2. Motivation
- 3. Apache Giraph • Inspired by Google’s Pregel but runs on Hadoop • “Think like a vertex” • Maximum value vertex example Processor 1 Processor 2 Time 5 5 5 5 2 5 5 5 2 1 5 5 2 1
- 4. Giraph on Hadoop / Yarn MapReduce YARN Giraph Hadoop 0.20.x Hadoop 0.20.203 Hadoop 2.0.x Hadoop 1.x
- 5. Send page rank value to neighbors for 30 iterations Calculate updated page rank value from neighbors Page rank in Giraph ! ! public class PageRankComputation extends BasicComputation<LongWritable, DoubleWritable, FloatWritable, DoubleWritable> { public void compute(Vertex<LongWritable, DoubleWritable, FloatWritable> vertex, Iterable<DoubleWritable> messages) { if (getSuperstep() >= 1) { double sum = 0; for (DoubleWritable message : messages) { sum += message.get(); } vertex.getValue().set(DoubleWritable((0.15d / getTotalNumVertices()) + 0.85d * sum); } if (getSuperstep() < 30) { sendMsgToAllEdges(new DoubleWritable(getVertexValue().get() / getNumOutEdges())); } else { voteToHalt(); } } }
- 6. Apache Giraph data flow Loading the graph Input format Split0 Split1 Split2 Split3 Master Load/ Send Graph Worker0 Load/ Send Graph Worker1 Storing the graph Worker0Worker1 Output format Part0 Part1 Part2 Part3 Part0 Part1 Part2 Part3 Compute / Iterate Compute/ Send Messages Compute/ Send Messages In-memory graph Part0 Part1 Part2 Part3Master Worker0Worker1 Sendstats/iterate!
- 7. Pipelined computation Master “computes” • Sets computation, in/out message, combiner for next super step •Can set/modify aggregator values Time Worker 0 Worker 1 Master phase 1a phase 1a phase 1b phase 1b phase 2 phase 2 phase 3 phase 3
- 8. Use case
- 9. Affinity propagation Frey and Dueck “Clustering by passing messages between data points” Science 2007 Organically discover exemplars based on similarity Initialization Intermediate Convergence
- 10. Responsibility r(i,k) • How well suited is k to be an exemplar for i? Availability a(i,k) • How appropriate for point i to choose point k as an exemplar given all of i’s responsibilities? Update exemplars • Based on known responsibilities/availabilities, which vertex should be my exemplar? ! * Dampen responsibility, availability 3 stages
- 11. Responsibility Every vertex i with an edge to k maintains responsibility of k for i Sends responsibility to k in ResponsibilityMessage (senderid, responsibility(i,k)) C A D B r(c,a) r(d,a) r(b,d) r(b,a)
- 12. Availability Vertex sums positive messages Sends availability to i in AvailabilityMessage (senderid, availability(i,k)) C A D B a(c,a) a(d,a) a(b,d) a(b,a)
- 13. Update exemplars Dampens availabilities and scans edges to find exemplar k Updates self-exemplar C A D Bupdate update update update exemplar=a exemplar=d exemplar=a exemplar=a
- 14. Master logic calculate responsibility calculate availability update exemplars initial state halt if (exemplars agree they are exemplars && changed exemplars < ∆) then halt, otherwise continue
- 15. Performance
- 16. Faster than Hive? Application Graph Size CPU Time Speedup Elapsed Time Speedup Page rank (single iteration) 400B+ edges 26x 120x Friends of friends score 71B+ edges 12.5x 48x
- 17. Apache Giraph scalability Scalability of workers (200B edges) Seconds 0 125 250 375 500 # of Workers 50 100 150 200 250 300 Giraph Ideal Scalability of edges (50 workers) Seconds 0 125 250 375 500 # of Edges 1E+09 7E+10 1E+11 2E+11 Giraph Ideal
- 18. Trillion social edges page rank Minutesperiteration 0 1 2 3 4 6/30/2013 6/2/2014 Improvements • GIRAPH-840 - Netty 4 upgrade • G1 Collector / tuning
- 20. Why balanced partitioning Random partitioning == good balance BUT ignores entity affinity 0 1 2 3 4 5 6 7 8 9 10 11
- 21. Balanced partitioning application Results from one service: Cache hit rate grew from 70% to 85%, bandwidth cut in 1/2 ! ! 0 2 3 5 6 9 11 1 4 7 8 10
- 22. Balanced label propagation results * Loosely based on Ugander and Backstrom. Balanced label propagation for partitioning massive graphs, WSDM '13
- 23. Partitioning experimentsSecondsperiteration 0 40 80 120 160 Random 47% Local Edges 345B edge page rank Improvements • 56% faster! • Native vertex remapping
- 24. Power-law graphs
- 25. Avoiding out-of-core Example: Mutual friends calculation between neighbors 1. Send your friends a list of your friends 2. Intersect with your friend list ! 1.23B (as of 1/2014) 200+ average friends (2011 S1) 8-byte ids (longs) = 394 TB / 100 GB machines 3,940 machines (not including the graph) A B C D E A:{D} D:{A,E} E:{D} B:{} C:{D} D:{C} A:{C} C:{A,E} E:{C} ! C:{D} D:{C} ! ! E:{}
- 27. Superstep splitting Subsets of sources/destinations edges per superstep * Currently manual - future work automatic! A Sources: A (on), B (off) Destinations: A (on), B (off) B B B A A A Sources: A (on), B (off) Destinations: A (off), B (on) B B B A A A Sources: A (off), B (on) Destinations: A (on), B (off) B B B A A A Sources: A (off), B (on) Destinations: A (off), B (on) B B B A A
- 28. Giraph in production Over 1.5 years in production Over 100 jobs processed a week 30+ applications in our internal application repository Sample production job - 700B+ edge
- 29. GiraphicJam demo
- 30. Giraph related projects Graft: The distributed Giraph debugger
- 31. Giraph roadmap 2/12 - 0.1 6/14 - 1.15/13 - 1.0
- 32. Future work Automatic checkpointing make scheduling more fair Investigate alternative computing models •Giraph++ (IBM research) •Giraphx (University at Buffalo, SUNY) Performance Lower the barrier to entry Applications