Titan: Scaling Graphs and TinkerPop3
11 likes5,766 views
The document discusses TitanDB, a scalable graph database designed for flexible data handling and optimized querying features. It outlines various concepts like vertex representation, graph partitioning, and querying optimizations, highlighting the advantages of using graph databases over traditional relational databases. Additionally, it provides examples of data models and queries, facilitating easier understanding of its application in real-world contexts.
![SELECT TOP (5) [t14].[ProductName]
FROM (SELECT COUNT(*) AS [value],
[t13].[ProductName]
FROM [customers] AS [t0]
CROSS APPLY (SELECT [t9].[ProductName]
FROM [orders] AS [t1]
CROSS JOIN [order details] AS [t2]
INNER JOIN [products] AS [t3]
ON [t3].[ProductID] = [t2].[ProductID]
CROSS JOIN [order details] AS [t4]
INNER JOIN [orders] AS [t5]
ON [t5].[OrderID] = [t4].[OrderID]
LEFT JOIN [customers] AS [t6]
ON [t6].[CustomerID] = [t5].[CustomerID]
CROSS JOIN ([orders] AS [t7]
CROSS JOIN [order details] AS [t8]
INNER JOIN [products] AS [t9]
ON [t9].[ProductID] = [t8].[ProductID])
WHERE NOT EXISTS(SELECT NULL AS [EMPTY]
FROM [orders] AS [t10]
CROSS JOIN [order details] AS [t11]
INNER JOIN [products] AS [t12]
ON [t12].[ProductID] = [t11].[ProductID]
WHERE [t9].[ProductID] = [t12].[ProductID]
AND [t10].[CustomerID] = [t0].[CustomerID]
AND [t11].[OrderID] = [t10].[OrderID])
AND [t6].[CustomerID] <> [t0].[CustomerID]
AND [t1].[CustomerID] = [t0].[CustomerID]
AND [t2].[OrderID] = [t1].[OrderID]
AND [t4].[ProductID] = [t3].[ProductID]
AND [t7].[CustomerID] = [t6].[CustomerID]
AND [t8].[OrderID] = [t7].[OrderID]) AS [t13]
WHERE [t0].[CustomerID] = N'ALFKI'
GROUP BY [t13].[ProductName]) AS [t14]
ORDER BY [t14].[value] DESC](https://image.slidesharecdn.com/seattlegraphdecember2014-141203131551-conversion-gate01/85/Titan-Scaling-Graphs-and-TinkerPop3-15-320.jpg)
![g.V.has('customerId’,cid).as('customer')
.out('orderedProduct’).as('products')
.in('orderedProduct').except('customer')
.out('orderedProduct’).except('products')
.productName.groupCount.cap.
.map{it.sort{-it.value}}.next()[0..<5]
h.p://sql2gremlin.com/](https://image.slidesharecdn.com/seattlegraphdecember2014-141203131551-conversion-gate01/85/Titan-Scaling-Graphs-and-TinkerPop3-16-320.jpg)
![Vertex-Centric Index
! Create a vertex-centric index per edge label
or property key to optimize for retrieval
patterns
! Through Titan’s management API
! Supports multiple indexes
name: Hercules
surname: Mighty
label: user
time:24
bought
title: “Muscle building for beginners”
label: product
[bought, time, DESC]](https://image.slidesharecdn.com/seattlegraphdecember2014-141203131551-conversion-gate01/85/Titan-Scaling-Graphs-and-TinkerPop3-26-320.jpg)
Titan: Scaling Graphs and TinkerPop3
- 1. AURELIUS THINKAURELIUS.COM Titan:db Graphs at Scale #TitanDB Matthias Broecheler @mbroecheler December 2nd, MMXIV
- 2. software is eating the world
- 3. >;N; >LCP?H software is eating the world
- 5. Tabular Relational Flexibility Handling NoSQL
- 6. Tabular Relational Flexibility Handling Data Model Query Language NoSQL
- 7. Tabular Relational NoSQL Graph Flexibility Handling Data Model Query Language Simplicity Flexibility Handling Data Model Query Language
- 8. productid: 52235 name: cup price: 12.55 userid: matt email: matt@ password: 12345 User Product
- 9. User Product userid email password ma. ma.@ 12345 john john@ qwerty billy billy@ abcde produc=d name price 52235 cup 12.55 42215 spoon 7.22 24529 knife 5.32
- 10. productid: 52235 name: cup price: 12.55 userid: matt email: matt@ password: 12345 User Product buy time: 9/5/14
- 11. User Product userid email password ma. ma.@ 12345 john john@ qwerty billy billy@ abcde produc=d name price 52235 cup 12.55 42215 spoon 7.22 24529 knife 5.32 userid produc=d =me ma. 52235 9/5/14 billy 42215 8/7/14 billy 42215 8/7/14 Buy
- 13. 7B;N >C> -;NN <OSg g.V.has(‘email’,’matt@’).out(‘buy’).valueMap!
- 14. 7B;N >C> -;NN <OSg SELECT Product.* ! FROM User! INNER JOIN Buy ! !ON User.userid = Buy.userid! INNER JOIN Product ! !ON Buy.productid = Product.productid! WHERE User.email = ‘matt@’ !
- 15. SELECT TOP (5) [t14].[ProductName] FROM (SELECT COUNT(*) AS [value], [t13].[ProductName] FROM [customers] AS [t0] CROSS APPLY (SELECT [t9].[ProductName] FROM [orders] AS [t1] CROSS JOIN [order details] AS [t2] INNER JOIN [products] AS [t3] ON [t3].[ProductID] = [t2].[ProductID] CROSS JOIN [order details] AS [t4] INNER JOIN [orders] AS [t5] ON [t5].[OrderID] = [t4].[OrderID] LEFT JOIN [customers] AS [t6] ON [t6].[CustomerID] = [t5].[CustomerID] CROSS JOIN ([orders] AS [t7] CROSS JOIN [order details] AS [t8] INNER JOIN [products] AS [t9] ON [t9].[ProductID] = [t8].[ProductID]) WHERE NOT EXISTS(SELECT NULL AS [EMPTY] FROM [orders] AS [t10] CROSS JOIN [order details] AS [t11] INNER JOIN [products] AS [t12] ON [t12].[ProductID] = [t11].[ProductID] WHERE [t9].[ProductID] = [t12].[ProductID] AND [t10].[CustomerID] = [t0].[CustomerID] AND [t11].[OrderID] = [t10].[OrderID]) AND [t6].[CustomerID] <> [t0].[CustomerID] AND [t1].[CustomerID] = [t0].[CustomerID] AND [t2].[OrderID] = [t1].[OrderID] AND [t4].[ProductID] = [t3].[ProductID] AND [t7].[CustomerID] = [t6].[CustomerID] AND [t8].[OrderID] = [t7].[OrderID]) AS [t13] WHERE [t0].[CustomerID] = N'ALFKI' GROUP BY [t13].[ProductName]) AS [t14] ORDER BY [t14].[value] DESC
- 16. g.V.has('customerId’,cid).as('customer') .out('orderedProduct’).as('products') .in('orderedProduct').except('customer') .out('orderedProduct’).except('products') .productName.groupCount.cap. .map{it.sort{-it.value}}.next()[0..<5] h.p://sql2gremlin.com/
- 17. TinkerPop 3 coming soon Graph Server Graph Algorithms Object-Graph Mapper Traversal Language Dataflow Processing Generic Graph API http://www.tinkerpop.com/docs/3.0.0.M6/
- 18. label: demigod name: Hercules label: god name: Neptune age: 4500 brothermother label: god name: Jupiter label: monster name: Cerberus father father brother battled pet time:12 label: god name: Pluto age: 4000 label: human name: Alcmene age: 45 label: titan name: Saturn age: 10000 label: monster name: Hydra battled time: 2
- 19. AURELIUS THINKAURELIUS.COM Titan:db Architecture & Internals titandb.io
- 21. Flexible Persistence Partitionability ConsistencyAvailability
- 22. Graph Schema ! Labels & Keys ! Vertex Property Cardinality ! Single, Set, List ! Edge Multiplicity ! Multi, Simple, Many2One, One2Many, One2One ! Constraints & Consistency
- 25. Vertex Representation 5 Property Property Out-Edge In-Edge Out-Edge In-Edge In-Edge LIQ CH>C=?M @IL @;MN P?LN?R =?HNLC= KO?LC?M <SN? IL>?L MILNCHA =?FF t =IFOGH q P;FO? LIQ E?S
- 26. Vertex-Centric Index ! Create a vertex-centric index per edge label or property key to optimize for retrieval patterns ! Through Titan’s management API ! Supports multiple indexes name: Hercules surname: Mighty label: user time:24 bought title: “Muscle building for beginners” label: product [bought, time, DESC]
- 27. battled battled battled time: 3 time: 5 v time: 1 fatherfought fought mother battled time: 9 v!
- 28. battled battled battled time: 3 time: 5 v time: 1 mother father battled time: 9 v.outE()!
- 29. battled battled battled time: 3 time: 5 v time: 1 battled time: 9 v.outE(‘battled’)!
- 30. v battled battled time: 1 time: 3 v.outE(‘battled’)! .has(‘time’,lt,5)! .inV!
- 31. Faster Querying ! Multi-Query ! Bulk querying reduces latency ! Currently manual ! Order in vertex-centric query ! Adjacency-constraint ! Faster edge retrievals " Titan Query Optimizer
- 32. Graph Index ! Composite internal index ! Can be unique ! Label specific index ! Multiple external indexes label: user name: Hercules surname: Mighty time:24 bought label: product title: “Muscle building for beginners”
- 33. g.E.has(‘location’,WITHIN, Geoshape.circle(38,24,50)! Full text & Geo Search
- 34. Token Ring (BOP) Graph Partitioning - assigns ids to map vertices into “optimal” token range - Maintains virtual partitions Vertical Partitioning = divide communities
- 35. Vertex Partitioning Each partition contains one “part” of a partitioned vertex Horizontal Partitioning = split super vertices
- 37. label: demigod name: Hercules label: god name: Neptune age: 4500 brothermother label: god name: Jupiter label: monster name: Cerberus father father brother battled pet time:12 label: god name: Pluto age: 4000 label: human name: Alcmene age: 45 label: titan name: Saturn age: 10000 label: monster name: Hydra battled time: 2 OLTP
- 38. label: demigod name: Hercules label: god name: Neptune age: 4500 brothermother label: god name: Jupiter label: monster name: Cerberus father father brother battled pet time:12 label: god name: Pluto age: 4000 label: human name: Alcmene age: 45 label: titan name: Saturn age: 10000 label: monster name: Hydra battled time: 2 OLAP
- 39. Faunus Work Flow g.V.out .out .count() hdfs://user/ubuntu/ output/job-0/ output/job-1/ output/job-2/ { graph* sideeffect* Compressed HDFS Graphs ! stored in sequence files ! variable length encoding ! prefix compression
- 40. Degree Distribution GitHub Network g.V.sideEffect{ it.degree = it.out(‘follows’).count() }.degree.groupCount
- 41. Degree Distribution P(k) ~ k-γ γ = 2.2
- 42. Complete Architecture ! Users & Apps Processing Event OLTP OLAP Q Continuous Computing Interactive Query Batch Query (ETL)
- 43. Use Cases
- 46. http://arli.us/edu-planet-scale © Sean York @ Pearson Education
- 47. http://arli.us/musicgraphintro Music Graph Knowledge Graph
- 49. 7B?H NI OM? 'L;JB $;N;<;M?Mg Map+ NoSQL K V V V V Persistence Tabular Data-driven Graph
- 50. 7B?H NI OM? 'L;JB $;N;<;M?Mg Map+ NoSQL K V V V V Persistence Tabular Data-driven Graph Complexity
- 51. AURELIUS THINKAURELIUS.COM @AURELIUSGRAPHS aureliusgraphs@googlegroups.com