GraphFrames: Graph Queries in Spark SQL by Ankur Dave
20 likes7,058 views
GraphFrames provides a unified API for graph queries and algorithms in Spark SQL. It translates graph patterns and algorithms to relational operations optimized by the Spark SQL query optimizer. Materializing the right views, such as the triplet view for GraphX algorithms or user-defined views for queries, can improve performance. An evaluation shows GraphFrames outperforms Neo4j for unanchored queries and approaches GraphX performance for graph algorithms using Spark SQL. Future work includes automatically suggesting optimal views and exploiting attribute-based partitioning.
![Graph Algorithms vs. Graph Queries
Graph Algorithm: PageRank
// Iterate until convergence
wikipedia.pregel(
sendMsg = { e =>
e.sendToDst(e.srcRank * e.weight)
},
mergeMsg = _ + _,
vprog = { (id, oldRank, msgSum) =>
0.15 + 0.85 * msgSum
})
Graph Query: Wikipedia Collaborators
wikipedia.find(
"(u1)-[e11]->(article1);
(u2)-[e21]->(article1);
(u1)-[e12]->(article2);
(u2)-[e22]->(article2)")
.select(
"*",
"e11.date – e21.date".as("d1"),
"e12.date – e22.date".as("d2"))
.sort("d1 + d2".desc).take(10)](https://image.slidesharecdn.com/2mhankurdave-160224032415/85/GraphFrames-Graph-Queries-in-Spark-SQL-by-Ankur-Dave-5-320.jpg)
GraphFrames: Graph Queries in Spark SQL by Ankur Dave
- 1. GraphFrames: Graph Queries in Spark SQL Ankur Dave UC BerkeleyAMPLab Joint work with Alekh Jindal (Microsoft), Li ErranLi (Uber), Joseph Gonzalez(UC Berkeley),Matei Zaharia (MIT and Databricks)
- 2. + Graph Queries 2016 Spark + GraphFrames Support for Graph Analysis in Spark + Graph Algorithms 2013 Spark + GraphX Relational Queries 2009 Spark Graph Updates + Anchored Traversals Neo4j, etc.
- 3. Graph Algorithms vs. Graph Queries ≈ x PageRank Alternating Least Squares Graph Algorithms Graph Queries
- 4. Graph Algorithms vs. Graph Queries Graph Algorithm: PageRank Graph Query: Wikipedia Collaborators Editor 1 Editor 2 Article 1 Article 2 ⇓ Article 1 Article 2 Editor 1 Editor 2 same day} same day}
- 5. Graph Algorithms vs. Graph Queries Graph Algorithm: PageRank // Iterate until convergence wikipedia.pregel( sendMsg = { e => e.sendToDst(e.srcRank * e.weight) }, mergeMsg = _ + _, vprog = { (id, oldRank, msgSum) => 0.15 + 0.85 * msgSum }) Graph Query: Wikipedia Collaborators wikipedia.find( "(u1)-[e11]->(article1); (u2)-[e21]->(article1); (u1)-[e12]->(article2); (u2)-[e22]->(article2)") .select( "*", "e11.date – e21.date".as("d1"), "e12.date – e22.date".as("d2")) .sort("d1 + d2".desc).take(10)
- 6. Separate Systems Graph Algorithms Graph Queries
- 7. Raw Wikipedia < / >< / >< / > XML Text Table Edit Graph Edit Table Frequent Collaborators Problem: Mixed Graph Analysis Hyperlinks PageRank User Product User Article Vandalism Suspects User User User Article
- 8. Solution: GraphFrames Graph Algorithms Graph Queries Spark SQL GraphFramesAPI Pattern Query Optimizer
- 9. GraphFrames API • Unifies graph algorithms, graph queries, and DataFrames • Available in Scala and Python class GraphFrame { def vertices: DataFrame def edges: DataFrame def find(pattern: String): DataFrame def registerView(pattern: String, df: DataFrame): Unit def degrees(): DataFrame def pageRank(): GraphFrame def connectedComponents(): GraphFrame ... }
- 10. Implementation Parsed Pattern Logical Plan Materialized Views Optimized Logical Plan DataFrame Result Query String Graph–Relational Translation Join Elimination and Reordering Spark SQL View Selection Graph Algorithms GraphX
- 11. Graph–Relational Translation B D A C Existing Logical Plan Output: A,B,C Src Dst ⋈C=Src Edge Table ID Attr VertexTable ⋈D=ID
- 12. Materialized View Selection GraphX: Triplet view enables efficient message-passing algorithms Vertices B A C D Edges A B A C B C C D A B Triplet View A C B C C D Graph + Updated PageRanks B A C D A
- 13. Materialized View Selection GraphFrames: User-defined views enable efficient graph queries Vertices B A C D Edges A B A C B C C D A B Triplet View A C B C C D Graph User-Defined Views PageRank Community Detection … Graph Queries
- 14. Join Elimination Src Dst 1 2 1 3 2 3 2 5 Edges ID Attr 1 A 2 B 3 C 4 D Vertices SELECT src, dst, attr AS src_attr FROM edges INNER JOIN vertices ON src = id; Standard vertex-edgejoin: SELECT src, dst FROM edges INNER JOIN vertices ON src = id; Unnecessaryjoin can be eliminated if tables satisfy referential integrity, simplifying graph–relational translation
- 15. Join Reordering A → B B → A ⋈A, B B → D C → B⋈B B → E⋈B C → D⋈B C → E⋈C, D ⋈C, E Example Query Left-Deep Plan BushyPlan A → B B → A ⋈A, B B → D C → B ⋈B B → E⋈B ⋈B ⋈B, C User-Defined View
- 16. Evaluation Faster than Neo4j for unanchored patternqueries 0 0.5 1 1.5 2 2.5 GraphFrames Neo4j Querylatency,s AnchoredPatternQuery 0 10 20 30 40 50 60 70 80 GraphFrames Neo4j Querylatency,s UnanchoredPatternQuery Triangle query on 1M edge subgraph of web-Google. Each system configured touse a single core.
- 17. Evaluation Approaches performance of GraphX for graph algorithms using Spark SQL whole-stage code generation 0 1 2 3 4 5 6 7 GraphFrames GraphX Naïve Spark Per-iterationruntime,s PageRankPerformance Per-iteration performance on web-Google, single 8-core machine. Naïve SparkusesScala RDD API.
- 18. Evaluation Registering the right views cangreatlyimprove performance
- 19. Future Work • Suggest views automatically • Exploit attribute-based partitioning in optimizer • Code generationfor single node
- 20. Try It Out! Preview available for Spark 1.4+ at: https://github.com/graphframes/graphframes Thanks to Databricks contributors Joseph Bradley,Xiangrui Meng,and Timothy Hunter. Watch for the release on Spark Packages in the coming weeks. ankurd@eecs.berkeley.edu