Big Graph Data
22 likes18,352 views
The document discusses Titan, a graph database designed for handling large-scale graph data, highlighting its features like real-time traversals, dynamic scalability, and variable consistency models. It details the architecture, data management strategies, query optimization techniques, and performance evaluations using benchmarks such as simulated social networking applications. Additionally, it covers integration with Hadoop for graph analytics and the future development goals for the Titan and Faunus ecosystems.
![Setup
$ ./titan-0.1.0/bin/gremlin.sh!
! ! !,,,/!
(o o)!
-----oOOo-(_)-oOOo-----!
gremlin> g = TitanFactory.open('/tmp/titan')!
==>titangraph[local:/tmp/titan]!](https://image.slidesharecdn.com/biggraphdatanov2012-121114220129-phpapp02/85/Big-Graph-Data-30-320.jpg)
![Create & Retrieve
gremlin> v = g.addVertex([username: ‘okram’])!
==>v[4]!
gremlin> v.map!
==>{username=okram}!
gremlin> g.V('username','okram')!
==>v[4]!](https://image.slidesharecdn.com/biggraphdatanov2012-121114220129-phpapp02/85/Big-Graph-Data-35-320.jpg)
![Recommendation
Engine
gremlin> v.out('pushed').out('to')[0..9].!
! ! ! in('to').in('pushed')[0..500].!
! ! ! except([v]).name.!
! ! ! groupCount.cap.next().sort{-it.value}[0..4]!](https://image.slidesharecdn.com/biggraphdatanov2012-121114220129-phpapp02/85/Big-Graph-Data-45-320.jpg)
![Recommendation
Engine
gremlin> v.out('pushed').out('to')[0..9].!
! ! ! in('to').in('pushed')[0..500].!
! ! ! except([v]).name.!
! ! ! groupCount.cap.next().sort{-it.value}[0..4]!
v = g.V(‘username’,’okram’):!
==>lvca=175!
==>spmallette=56!
==>sgomezvillamor=36!
==>mbroecheler=33!
==>joshsh=20!](https://image.slidesharecdn.com/biggraphdatanov2012-121114220129-phpapp02/85/Big-Graph-Data-46-320.jpg)
![Recommendation
Engine
gremlin> v.out('pushed').out('to')[0..9].!
! ! ! in('to').in('pushed')[0..500].!
! ! ! except([v]).name.!
! ! ! groupCount.cap.next().sort{-it.value}[0..4]!
v = g.V(‘username’,’torvalds’):!
==>iksaif=90!
==>rjwysocki=22!
==>kernel-digger=20!
==>giuseppecalderaro=16!
==>groeck=15!](https://image.slidesharecdn.com/biggraphdatanov2012-121114220129-phpapp02/85/Big-Graph-Data-47-320.jpg)
![Faunus Setup
$ bin/gremlin.sh !
,,,/!
(o o)!
-----oOOo-(_)-oOOo-----!
gremlin> g = FaunusFactory.open('bin/titan-hbase.properties')!
==>faunusgraph[titanhbaseinputformat]!
gremlin> g.getProperties()!
==>faunus.graph.input.format=com.thinkaurelius.faunus.formats.titan.hbase.TitanHBaseInputFormat
==>faunus.graph.output.format=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat!
==>faunus.sideeffect.output.format=org.apache.hadoop.mapreduce.lib.output.TextOutputFormat!
==>faunus.output.location=dbpedia!
==>faunus.output.location.overwrite=true!
gremlin> g._() !
12/11/09 15:17:45 INFO mapreduce.FaunusCompiler: Compiled to 1 MapReduce job(s)!
12/11/09 15:17:45 INFO mapreduce.FaunusCompiler: Executing job 1 out of 1:
MapSequence[com.thinkaurelius.faunus.mapreduce.transform.IdentityMap.Map]!
12/11/09 15:17:50 INFO mapred.JobClient: Running job: job_201211081058_0003!](https://image.slidesharecdn.com/biggraphdatanov2012-121114220129-phpapp02/85/Big-Graph-Data-55-320.jpg)
Big Graph Data
- 1. BIG GRAPH DATA Understanding a Complex World Matthias Broecheler, CTO @mbroecheler AURELIUS November XIII, MMXII THINKAURELIUS.COM
- 2. I Graph Foundation AURELIUS THINKAURELIUS.COM
- 3. name: Neptune name: Alcmene type: god type: god Vertex Property name: Saturn name: Jupiter name: Hercules type: titan type: god type: demigod name: Pluto name: Cerberus type: god type: monster Graph
- 4. name: Neptune name: Alcmene type: god type: god Edge brother mother name: Saturn name: Jupiter name: Hercules type: titan type: god type: demigod father father Edge battled brother Property time:12 name: Pluto name: Cerberus type: god type: monster Edge Type pet Graph
- 5. name: Neptune name: Alcmene type: god type: god brother mother name: Saturn name: Jupiter name: Hercules type: titan type: god type: demigod father father battled brother time:12 name: Pluto name: Cerberus type: god type: monster pet Path
- 6. name: Neptune name: Alcmene type: god type: god brother mother name: Saturn name: Jupiter name: Hercules type: titan type: god type: demigod father father battled brother time:12 name: Pluto name: Cerberus type: god type: monster pet Degree
- 7. I Connected World AURELIUS THINKAURELIUS.COM
- 8. HEALTH
- 9. HEALTH
- 10. HEALTH
- 11. HEALTH
- 12. ECONOMY
- 13. ECONOMY
- 14. ECONOMY
- 15. ECONOMY
- 16. Social Systems
- 17. Social Systems
- 18. Social Systems
- 19. Social Systems
- 23. III Titan Graph Database AURELIUS THINKAURELIUS.COM
- 24. Titan Features Numerous Concurrent Users Many Short Transactions read/write Real-time Traversals (OLTP) High Availability Dynamic Scalability Variable Consistency Model ACID or eventual consistency Real-time Big Graph Data
- 25. Storage Backends Partitionability Consistency Availability
- 26. Titan Features I. Data Management II. Vertex-Centric Indices
- 27. Titan Features III. Graph Partitioning IV. Edge Compression
- 28. Titan Ecosystem Native Blueprints Graph Server Implementation Graph Gremlin Query Algorithms Language Object-Graph Mapper Rexster Server Traversal Language any Titan graph can be exposed as a REST endpoint Dataflow Processing Generic Graph API
- 29. IV Github Network AURELIUS THINKAURELIUS.COM
- 30. Setup $ ./titan-0.1.0/bin/gremlin.sh! ! ! !,,,/! (o o)! -----oOOo-(_)-oOOo-----! gremlin> g = TitanFactory.open('/tmp/titan')! ==>titangraph[local:/tmp/titan]!
- 31. Titan Storage Model Adjacency list in one 5 column family Row key = vertex id Each property and edge in one column 5 Denormalized, i.e. stored twice Direction and label/key as column prefix Use slice predicate for quick retrieval
- 32. created USER edited opened pushed COMMENT PAGE on ISSUE COMMIT on on to in REPOSITORY
- 33. Defining Property Keys gremlin> g.makeType().name(‘username’).! ! ! ! dataType(String.class).! ! ! ! functional().! ! ! ! indexed().unique().! ! ! ! makePropertyKey()! gremlin> g.makeType().name(‘time’).! ! ! ! dataType(Long.class).! ! ! ! functional().makePropertyKey()!
- 34. Defining Edge Labels gremlin> g.makeType().name(‘on’).! ! ! ! makeEdgeLabel()! gremlin> g.makeType().name(‘pushed’).! ! ! ! primaryKey(time).! ! ! ! makeEdgeLabel()! gremlin> g.makeType().name(‘in’).! ! ! ! unidirected().! ! ! ! makeEdgeLabel()!
- 35. Create & Retrieve gremlin> v = g.addVertex([username: ‘okram’])! ==>v[4]! gremlin> v.map! ==>{username=okram}! gremlin> g.V('username','okram')! ==>v[4]!
- 36. Titan Locking Locking ensures consistency when it is needed name : Hercules 5 Titan uses time stamped consistent reads and writes 9 on separate CFs for locking Uses name : Property uniqueness: .unique() name : Hercules Jupiter Functional edges: .functional() father Global ID management x name : father Pluto
- 37. Titan Indexing Vertices can be retrieved by property key + value name : Hercules 5 Titan maintains index in a separate column family as name : Jupiter 9 graph is updated Only need to define a property key as .index()
- 38. Basic Queries gremlin> v.out(‘pushed’)! gremlin> v.out(‘pushed’).out(‘to’).name! gremlin> v.out(‘pushed’).out(‘to’).dedup.name! gremlin> v.out(‘pushed’).out(‘to’).dedup.! ! ! ! name.sort{it}! gremlin> v.outE(‘pushed’).has(‘time’,T.gt,1000).inV!
- 39. Basic Queries gremlin> v.out(‘pushed’)! gremlin> v.out(‘pushed’).out(‘to’).name! gremlin> v.out(‘pushed’).out(‘to’).dedup.name! gremlin> v.out(‘pushed’).out(‘to’).dedup.! ! ! ! name.sort{it}! gremlin> v.outE(‘pushed’).has(‘time’,T.gt,1000).inV! Query Optimization
- 40. Vertex-Centric Indices Sort and index edges per vertex by primary key Primary key can be composite Enables efficient focused traversals Only retrieve edges that matter Uses push down predicates for quick, index-driven retrieval
- 41. battled battled battled time: 1 time: 3 time: 5 mother battled v v.query()! time: 9 father fought fought
- 42. battled battled battled time: 1 time: 3 time: 5 mother battled v v.query()! time: 9 .direction(OUT)! father
- 43. battled battled battled time: 1 time: 3 time: 5 battled v v.query()! time: 9 .direction(OUT)! .labels(‘battled’)!
- 44. battled battled time: 1 time: 3 v v.query()! .direction(OUT)! .labels(‘battled’)! .has(‘time,T.lt,5)!
- 45. Recommendation Engine gremlin> v.out('pushed').out('to')[0..9].! ! ! ! in('to').in('pushed')[0..500].! ! ! ! except([v]).name.! ! ! ! groupCount.cap.next().sort{-it.value}[0..4]!
- 46. Recommendation Engine gremlin> v.out('pushed').out('to')[0..9].! ! ! ! in('to').in('pushed')[0..500].! ! ! ! except([v]).name.! ! ! ! groupCount.cap.next().sort{-it.value}[0..4]! v = g.V(‘username’,’okram’):! ==>lvca=175! ==>spmallette=56! ==>sgomezvillamor=36! ==>mbroecheler=33! ==>joshsh=20!
- 47. Recommendation Engine gremlin> v.out('pushed').out('to')[0..9].! ! ! ! in('to').in('pushed')[0..500].! ! ! ! except([v]).name.! ! ! ! groupCount.cap.next().sort{-it.value}[0..4]! v = g.V(‘username’,’torvalds’):! ==>iksaif=90! ==>rjwysocki=22! ==>kernel-digger=20! ==>giuseppecalderaro=16! ==>groeck=15!
- 48. Titan Embedding Rexster RexPro lightweight Gremlin Server based on Grizzly Titan Gremlin Engine Embedded Storage Backend in-JVM method calls
- 49. Graph Partitioning Goal: Vertex Co-location Titan maintains multiple ID Pools Ordered Partitioner in Storage Backend Dynamically determines optimal partition and allocates corresponding ID Pool IDs
- 50. What’s coming Full-text indexing external index system integration Bulk Loading integration with storage backend utilities and Hadoop ingestion 240 Billion Edge Benchmark performance analysis and improvements across the entire stack
- 51. V Faunus Graph Analytics AURELIUS THINKAURELIUS.COM
- 52. Faunus Features Hadoop-based Graph Computing Framework Graph Analytics Breadth-first Traversals Global Graph Computations Batch Big Graph Data
- 53. Faunus Architecture g._()!
- 54. Faunus Work Flow g.V.out .out .count() hdfs://user/ubuntu/ output/job-0/ output/job-1/ graph* output/job-2/ { sideeffect* Compressed HDFS Graphs stored in sequence files variable length encoding prefix compression
- 55. Faunus Setup $ bin/gremlin.sh ! ,,,/! (o o)! -----oOOo-(_)-oOOo-----! gremlin> g = FaunusFactory.open('bin/titan-hbase.properties')! ==>faunusgraph[titanhbaseinputformat]! gremlin> g.getProperties()! ==>faunus.graph.input.format=com.thinkaurelius.faunus.formats.titan.hbase.TitanHBaseInputFormat ==>faunus.graph.output.format=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat! ==>faunus.sideeffect.output.format=org.apache.hadoop.mapreduce.lib.output.TextOutputFormat! ==>faunus.output.location=dbpedia! ==>faunus.output.location.overwrite=true! gremlin> g._() ! 12/11/09 15:17:45 INFO mapreduce.FaunusCompiler: Compiled to 1 MapReduce job(s)! 12/11/09 15:17:45 INFO mapreduce.FaunusCompiler: Executing job 1 out of 1: MapSequence[com.thinkaurelius.faunus.mapreduce.transform.IdentityMap.Map]! 12/11/09 15:17:50 INFO mapred.JobClient: Running job: job_201211081058_0003!
- 56. Graph Analytics gremlin> g.E.has('label',’followed').keep.! ! ! !V.sideEffect('{it.degree = it.outE.count()}').! ! ! !degree.groupCount! gremlin> g.E.has('label','pushed').keep.! ! ! !V.sideEffect('{it.degree = it.outE.count()}').! ! ! !degree.groupCount!
- 58. Follow Degree Distribution P(k) ~ k-γ γ = 2.2
- 60. Global Recommendations gremlin> g.E.has('label','pushed','to').keep.! ! ! !V.out('pushed').out('to').! ! ! !in('to').in('pushed').! ! ! !sideEffect('{it.score =it.pathCounter}').! ! ! !score.order(F.decr,'name')! # Top 5:! Jippi ! ! ! !60892182927! garbear ! ! !30095282886! FakeHeal ! ! !30038040349! brianchandotcom !24684133382! nyarla ! ! !15230275746!
- 61. What’s coming Faunus 0.1 Bulk Loading loaded graph into Titan loading derivations into Titan Extending Gremlin Support currently only a subset is of Gremlin implemented Operational Tools
- 62. I Graph = Relationship Centric
- 63. II Graph = Agile Data Model
- 64. III Graph = Algebraic Data Model
- 65. Aurelius Graph Cluster Apache 2 Map/Reduce Load & Compress Analysis results back into Titan Stores a massive-scale Batch processing of large Runs global graph algorithms property graph allowing real- graphs with Hadoop on large, compressed, time traversals and updates in-memory graphs
- 66. Speed of Traversal/Process The Graph Landscape Illustration only, not to scale Size of Graph
- 67. TINKERPOP.COM
- 68. Thanks! Vadas Gintautas Marko Rodriguez @vadasg @twarko Stephen Mallette Daniel LaRocque @spmallette AURELIUS THINKAURELIUS.COM
- 70. XVX Benchmark Results AURELIUS THINKAURELIUS.COM
- 71. XVX - I Titan Performance Evaluation on Twitter-like Benchmark AURELIUS THINKAURELIUS.COM
- 72. Twitter Benchmark 1.47 billion followship edges and 41.7 million users Loaded into Titan using BatchGraph Twitter in 2009, crawled by Kwak et. al 4 Transaction Types Create Account (1%) Publish tweet (15%) Read stream (76%) Recommendation (8%) Follow recommended user (30%) Kwak, H., Lee, C., Park, H., Moon, S., “What is Twitter, a Social Network or a News Media?,” World Wide Web Conference, 2010.
- 73. Benchmark Setup 6 cc1.4xl Cassandra nodes in one placement group Cassandra 1.10 40 m1.small worker machines repeatedly running transactions simulating servers handling user requests EC2 cost: $11/hour
- 74. Benchmark Results Transaction Type Number of tx Mean tx time Std of tx time Create account 379,019 115.15 ms 5.88 ms Publish tweet 7,580,995 18.45 ms 6.34 ms Read stream 37,936,184 6.29 ms 1.62 ms Recommendation 3,793,863 67.65 ms 13.89 ms Total 49,690,061 Runtime 2.3 hours 5,900 tx/sec
- 75. Peak Load Results Transaction Type Number of tx Mean tx time Std of tx time Create account 374,860 172.74 ms 10.52 ms Publish tweet 7,517,667 70.07 ms 19.43 ms Read stream 37,618,648 24.40 ms 3.18 ms Recommendation 3,758,266 229.83 ms 29.08 ms Total 49,269,441 Runtime 1.3 hours 10,200 tx/sec
- 76. Benchmark Conclusion Titan can handle 10s of thousands of concurrent users with short response 5mes even for complex traversals on a simulated social networking applica5on based on real-‐world network data with billions of edges and millions of users in a standard EC2 deployment. For more informa5on on the benchmark: hDp://thinkaurelius.com/2012/08/06/5tan-‐provides-‐real-‐5me-‐big-‐graph-‐ data/