Handling Billions of Edges in a Graph Database
3 likes3,309 views
The document discusses the capabilities and features of ArangoDB, a multi-model database that supports graph databases. It explains typical and non-typical graph queries, traversal techniques, and complexities involved in querying, as well as the challenges of distributing graphs in a cluster setup. Additionally, it mentions the use of index-free adjacency for efficient edge management and explores domain-based distribution for natural clustering of data.
![AQL
13
FOR user IN users
FILTER user.name == "alice"
FOR recommendation, action, path IN 3 ANY user has_bought
FILTER path.vertices[2].age <= user.age + 5
AND path.vertices[2].age >= user.age - 5
FILTER recommendation.price < 25
LIMIT 10
RETURN recommendation
Alice TV
has_bought
Bob Playstation
has_boughthas_bought
alice.age - 5 <= bob.age &&
bob.age <= alice.age + 5 playstation.price < 25](https://image.slidesharecdn.com/handlingbillionsofedgesinagraphdatabase-michaelhackstein-161005144228/85/Handling-Billions-of-Edges-in-a-Graph-Database-34-320.jpg)
Handling Billions of Edges in a Graph Database
- 1. www.arangodb.com Handling Billions Of Edges in a Graph Database Michael Hackstein @mchacki New Technology
- 2. Michael Hackstein ‣ ArangoDB Core Team ‣ Web Frontend ‣ Graph visualisation ‣ Graph features ‣ SmartGraphs ‣ Host of cologne.js ‣ Master’s Degree (spec. Databases and Information Systems) 2
- 3. What are Graph Databases ‣ Schema-free Objects (Vertices) ‣ Relations between them (Edges) ‣ Edges have a direction 3 { name: "alice", age: 32 } { name: "bob", age: 35, size: 1,73m } { name: "fishing" } { name: "reading" } { name: "dancing" } married hobby hobby hobby hobby
- 4. What are Graph Databases ‣ Schema-free Objects (Vertices) ‣ Relations between them (Edges) ‣ Edges have a direction ‣ Edges can be queried in both directions ‣ Easily query a range of edges (2 to 5) ‣ Undefined number of edges (1 to *) ‣ Shortest Path between two vertices 3 { name: "alice", age: 32 } { name: "bob", age: 35, size: 1,73m } { name: "fishing" } { name: "reading" } { name: "dancing" } married hobby hobby hobby hobby
- 6. Typical Graph Queries ‣ Give me all friends of Alice 4
- 7. Typical Graph Queries ‣ Give me all friends of Alice ‣ Give me all friends-of-friends of Alice 4
- 8. Typical Graph Queries ‣ Give me all friends of Alice ‣ Give me all friends-of-friends of Alice ‣ What is the linking path between Alice and Bob 4
- 9. Typical Graph Queries ‣ Give me all friends of Alice ‣ Give me all friends-of-friends of Alice ‣ What is the linking path between Alice and Bob ‣ Which Trainstations can I reach if I am allowed to drive a distance of 6 stations on my ticket 4
- 10. Typical Graph Queries ‣ Give me all friends of Alice ‣ Give me all friends-of-friends of Alice ‣ What is the linking path between Alice and Bob ‣ Which Trainstations can I reach if I am allowed to drive a distance of 6 stations on my ticket ‣ Pattern Matching: 4
- 11. Typical Graph Queries ‣ Give me all friends of Alice ‣ Give me all friends-of-friends of Alice ‣ What is the linking path between Alice and Bob ‣ Which Trainstations can I reach if I am allowed to drive a distance of 6 stations on my ticket ‣ Pattern Matching: ‣ Give me all users that share two hobbies with Alice 4
- 12. Typical Graph Queries ‣ Give me all friends of Alice ‣ Give me all friends-of-friends of Alice ‣ What is the linking path between Alice and Bob ‣ Which Trainstations can I reach if I am allowed to drive a distance of 6 stations on my ticket ‣ Pattern Matching: ‣ Give me all users that share two hobbies with Alice ‣ Give me all products that at least one of my friends has bought together with the products I already own, ordered by how many friends have bought it and the products rating, but only 20 of them. 4
- 14. Non-Typical Graph Queries ‣ Give me all users which have an age attribute between 21 and 35. 5
- 15. Non-Typical Graph Queries ‣ Give me all users which have an age attribute between 21 and 35. ‣ Give me the age distribution of all users 5
- 16. Non-Typical Graph Queries ‣ Give me all users which have an age attribute between 21 and 35. ‣ Give me the age distribution of all users ‣ Group all users by their name 5
- 17. Traversal 6 Iterate down two edges with some filters
- 18. Traversal ‣ We first pick a start vertex (S) 6 S Iterate down two edges with some filters
- 19. Traversal ‣ We first pick a start vertex (S) ‣ We collect all edges on S 6 S A B C Iterate down two edges with some filters
- 20. Traversal ‣ We first pick a start vertex (S) ‣ We collect all edges on S ‣ We apply filters on edges 6 S A B C Iterate down two edges with some filters
- 21. Traversal ‣ We first pick a start vertex (S) ‣ We collect all edges on S ‣ We apply filters on edges ‣ We iterate down one of the new vertices (A) 6 S A B C D E Iterate down two edges with some filters
- 22. Traversal ‣ We first pick a start vertex (S) ‣ We collect all edges on S ‣ We apply filters on edges ‣ We iterate down one of the new vertices (A) ‣ We apply filters on edges 6 S A B C D E Iterate down two edges with some filters
- 23. Traversal ‣ We first pick a start vertex (S) ‣ We collect all edges on S ‣ We apply filters on edges ‣ We iterate down one of the new vertices (A) ‣ We apply filters on edges ‣ The next vertex (E) is in desired depth. Return the path S -> A -> E 6 S A B C D E Iterate down two edges with some filters
- 24. Traversal ‣ We first pick a start vertex (S) ‣ We collect all edges on S ‣ We apply filters on edges ‣ We iterate down one of the new vertices (A) ‣ We apply filters on edges ‣ The next vertex (E) is in desired depth. Return the path S -> A -> E ‣ Go back to the next unfinished vertex (B) 6 S A B C D E Iterate down two edges with some filters
- 25. Traversal ‣ We first pick a start vertex (S) ‣ We collect all edges on S ‣ We apply filters on edges ‣ We iterate down one of the new vertices (A) ‣ We apply filters on edges ‣ The next vertex (E) is in desired depth. Return the path S -> A -> E ‣ Go back to the next unfinished vertex (B) ‣ We iterate down on (B) 6 S A B C D E Iterate down two edges with some filters F
- 26. Traversal ‣ We first pick a start vertex (S) ‣ We collect all edges on S ‣ We apply filters on edges ‣ We iterate down one of the new vertices (A) ‣ We apply filters on edges ‣ The next vertex (E) is in desired depth. Return the path S -> A -> E ‣ Go back to the next unfinished vertex (B) ‣ We iterate down on (B) ‣ We apply filters on edges 6 S A B C D E Iterate down two edges with some filters F
- 27. Traversal ‣ We first pick a start vertex (S) ‣ We collect all edges on S ‣ We apply filters on edges ‣ We iterate down one of the new vertices (A) ‣ We apply filters on edges ‣ The next vertex (E) is in desired depth. Return the path S -> A -> E ‣ Go back to the next unfinished vertex (B) ‣ We iterate down on (B) ‣ We apply filters on edges ‣ The next vertex (F) is in desired depth. Return the path S -> B -> F 6 S A B C D E Iterate down two edges with some filters F
- 28. Traversal - Complexity ‣ Once: ‣ Find the start vertex ‣ For every depth: ‣ Find all connected edges ‣ Filter non-matching edges ‣ Find connected vertices ‣ Filter non-matching vertices 7 Depends on indexes: Hash: Edge-Index or Index-Free: Linear in edges: Depends on indexes: Hash: Linear in vertices: Only one pass: O 1 1 n n * 1 n 3n
- 29. Traversal - Complexity ‣ Linear sounds evil? ‣ NOT linear in All Edges O(E) ‣ Only Linear in relevant Edges n < E ‣ Traversals solely scale with their result size. ‣ They are not effected at all by total amount of data ‣ BUT: Every depth increases the exponent: O(3 * n ) ‣ "7 degrees of separation": 3*n < E < 3*n 8 d 6 7
- 30. ‣ MULTI-MODEL database ‣ Stores Documents and Graphs ‣ Query language AQL ‣ Document Queries ‣ Graph Queries ‣ Joins ‣ All can be combined in the same statement ‣ ACID support including Multi Collection Transactions 9
- 31. AQL 10 FOR user IN users RETURN user
- 32. AQL 11 FOR user IN users FILTER user.name == "alice" RETURN user
- 33. AQL 12 FOR user IN users FILTER user.name == "alice" FOR product IN OUTBOUND user has_bought RETURN product Alice TV has_bought
- 34. AQL 13 FOR user IN users FILTER user.name == "alice" FOR recommendation, action, path IN 3 ANY user has_bought FILTER path.vertices[2].age <= user.age + 5 AND path.vertices[2].age >= user.age - 5 FILTER recommendation.price < 25 LIMIT 10 RETURN recommendation Alice TV has_bought Bob Playstation has_boughthas_bought alice.age - 5 <= bob.age && bob.age <= alice.age + 5 playstation.price < 25
- 36. First Boost - Vertex Centric Indices ‣ Remember Complexity? O(3 * n ) ‣ Filtering of non-matching edges is linear for every depth ‣ Index all edges based on their vertices and arbitrary other attributes ‣ Find initial set of edges in identical time ‣ Less / No post-filtering required ‣ This decreases the n 15 d
- 38. Scaling ‣ Vertex-Centric Indexes help with super-nodes ‣ But what if the graph is too large for one machine? ‣ Distribute graph on several machines (sharding) ‣ How to query it now? ‣ No global view of the graph possible any more ‣ What about edges between servers? 17
- 39. 18 First let's do the cluster thingy
- 40. 19
- 41. 20 Marathon
- 42. 21 Demo Time DC / OS
- 43. Is Mesosphere required? ‣ ArangoDB can run clusters without it ‣ Setup Requires manual effort (can be scripted): ‣ Configure IP addresses ‣ Correct startup ordering ‣ This works: ‣ Automatic Failover (Follower takes over if leader dies) ‣ Rebalancing of shards ‣ Everything inside of ArangoDB ‣ This is based on Mesos: ‣ Complete self healing ‣ Automatic restart of ArangoDBs (on new machines) ➡ We suggest you have someone on call 22
- 45. Dangers of Sharding ‣ Only parts of the graph on every machine ‣ Neighboring vertices may be on different machines ‣ Even edges could be on other machines than their vertices ‣ Queries need to be executed in a distributed way ‣ Result needs to be merged locally 24
- 46. Random Distribution ‣ Disadvantages: ‣ Neighbors on different machines ‣ Probably edges on other machines than their vertices ‣ A lot of network overhead is required for querying 25 ‣ Advantages: ‣ every server takes an equal portion of data ‣ easy to realize ‣ no knowledge about data required ‣ always works
- 47. Random Distribution ‣ Disadvantages: ‣ Neighbors on different machines ‣ Probably edges on other machines than their vertices ‣ A lot of network overhead is required for querying 25 ‣ Advantages: ‣ every server takes an equal portion of data ‣ easy to realize ‣ no knowledge about data required ‣ always works
- 48. Index-Free Adjacency 26 ‣ Used by most other graph databases ‣ Every vertex maintains two lists of it's edges (IN and OUT) ‣ Do not use an index to find edges ‣ How to shard this?
- 49. Index-Free Adjacency 26 ‣ Used by most other graph databases ‣ Every vertex maintains two lists of it's edges (IN and OUT) ‣ Do not use an index to find edges ‣ How to shard this?
- 50. Index-Free Adjacency 26 ‣ Used by most other graph databases ‣ Every vertex maintains two lists of it's edges (IN and OUT) ‣ Do not use an index to find edges ‣ How to shard this?
- 51. Index-Free Adjacency 26 ‣ Used by most other graph databases ‣ Every vertex maintains two lists of it's edges (IN and OUT) ‣ Do not use an index to find edges ‣ How to shard this? ????
- 52. Index-Free Adjacency ‣ ArangoDB uses an hash-based EdgeIndex (O(1) - lookup) ‣ The vertex is independent of it's edges ‣ It can be stored on a different machine 26 ‣ Used by most other graph databases ‣ Every vertex maintains two lists of it's edges (IN and OUT) ‣ Do not use an index to find edges ‣ How to shard this? ????
- 53. Domain Based Distribution ‣ Many Graphs have a natural distribution ‣ By country/region for People ‣ By tags for Blogs ‣ By category for Products ‣ Most edges in same group ‣ Rare edges between groups 27
- 54. Domain Based Distribution ‣ Many Graphs have a natural distribution ‣ By country/region for People ‣ By tags for Blogs ‣ By category for Products ‣ Most edges in same group ‣ Rare edges between groups 27
- 55. Domain Based Distribution ‣ Many Graphs have a natural distribution ‣ By country/region for People ‣ By tags for Blogs ‣ By category for Products ‣ Most edges in same group ‣ Rare edges between groups 27 uses Domain Knowledge for short-cuts
- 58. Thank you ‣ Further questions? ‣ Follow us on twitter: @arangodb ‣ Join our slack: slack.arangodb.com ‣ Follow me on twitter/github: @mchacki ‣ Write me a mail: michael@arangodb.com 30