Everything Goes Better With Bacon: Revisiting the Six Degrees Problem with a Graph Database
5 likes3,133 views
The document discusses the 'Six Degrees of Kevin Bacon' concept and its relation to graph databases, exploring the efficiency of graph analysis for connecting various entities in the film industry. It explains how graph databases differ from relational databases and emphasizes the need for appropriate tools for navigating and analyzing complex data. Furthermore, it addresses techniques like policy settings and graph views to manage supernodes, allowing for more efficient data queries within the InfiniteGraph database system.
![Six Degrees of Bacon
“…any individual involved in the Hollywood, California film industry
can be linked through his or her film roles to actor Kevin Bacon
within six steps”
[http://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon]
Gina Menza
Images Courtesy of IMDB (www.imdb.com)
A Tale of Two Kevins](https://image.slidesharecdn.com/igjaxconf2013sixdegrees-130611141145-phpapp01/85/Everything-Goes-Better-With-Bacon-Revisiting-the-Six-Degrees-Problem-with-a-Graph-Database-3-320.jpg)
Everything Goes Better With Bacon: Revisiting the Six Degrees Problem with a Graph Database
- 1. www.Objectivity.com Nick Quinn Lead Developer - InfiniteGraph
- 2. What are we talking about today? Not that Bacon This Bacon! • Intro to the Six Degrees Problem • What is a Graph Database? • Why Bacon in Graph Database? • How we solved the problem Images Courtesy of IMDB (www.imdb.com)
- 3. Six Degrees of Bacon “…any individual involved in the Hollywood, California film industry can be linked through his or her film roles to actor Kevin Bacon within six steps” [http://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon] Gina Menza Images Courtesy of IMDB (www.imdb.com) A Tale of Two Kevins
- 4. Why Six Degrees of Bacon? Actor Age # of Projects Kevin Bacon 54 76 Harrison Ford 70 70 Tom Cruise 50 40 Julia Roberts 45 50 Tom Hanks 56 73 Denzel Washington 58 53 Michael Caine 80 157 Kiefer Sutherland 46 82 Kevin Bacon Images Courtesy of IMDB (www.imdb.com)
- 5. Bacon Numbers in Google In the summer of 2012, Google started to allow users to find the bacon number of any actor simply by following his or her name with “bacon number”. Morgan Freeman The Dark Night Rises appeared in Gary Oldman appeared in Kevin Bacon Criminal Law appeared in appeared in www.google.com Graphical Representation
- 6. What is a Graph Database ?
- 7. The Physical Data Model • Difference between relational & graph databases Meetings P1 Place TimeP2 Alice Denver 5-27-10Bob Calls From Time DurationTo Bob 13:20 25Carlos Bob 17:10 15Charlie Payments From Date AmountTo Carlos 5-12-10 100000Charlie Met 5-27-10 Alice Called 13:20 Bob Paid 100000 Carlos Charlie Called 17:10 Rows/Columns/Tables Relationship/Graph Optimized
- 8. Connecting Data Person Building ? Work Live RR Visit Eat Shop
- 9. Who is Gina Menza? • How do we get meaning from highly connected data? Gina Menza Jury Forewoman Miss Jeffries Images Courtesy of IMDB (www.imdb.com)
- 10. Strength of Connections Matter! • Why 6 degrees of separation and not 3.74? • We need analysis tools in order to – identify and filter out “unimportant” data and – infer what needs to be filtered as we investigate it. “When considering another person in the world, a friend of your friend knows a friend of their friend” - facebook
- 11. Why Bacon in a Graph Database ?
- 12. Graph Analysis • Why use Graph Databases for graph analysis? – Dynamic on Live Data – Feedback/Inference – Optimized for concurrent user access – Handles big data problems – Native Graph Traversal API – Manage memory efficiently
- 13. Paths to Bacon Bacon Number (Degree of Separation / 2) # of People 1 2823 2 323677 3 1088560 4 272905 5 22533 6 2300 Using the IMDB (www.imdb.com) data set, we can study how many paths can be found by degrees of separation from Kevin Bacon. Out of 5,067,124 nodes and 11,505,797 edges, we get the following: 0 200000 400000 600000 800000 1000000 1200000 1 2 3 4 5 6 # of People
- 14. Big Data + Graph = Big Graph Data 4 Degrees of Kevin Bacon (Breadth First up to 20K connections) Images generated using the IG Visualizer
- 15. Analyzing Bacon • To be able to perform meaningful analysis, these are things that you will need: – Ingest IMDB Dataset – About 50 Formatted compressed files (Largest > 200 MB) – Custom algorithm support to perform meaningful analysis • Optimize queries to get results back in reasonable time – Visualization tool to test and view the results of the navigation (optional)
- 16. How IG Sizzles Your Bacon Ingest Update Navigate Massive graph data require efficient and intelligent tools to analyze and understand it.
- 17. Super Simple Java API Actor bacon = new Actor(“Kevin Bacon”); imdbGraphDB.addVertex( bacon ); Movie apollo= new Movie(“Apollo 13”, 1995); imdbGraphDB.addVertex( apollo ); ActedIn bacon2apollo = new ActedIn(“Jack Swigert”); imdbGraphDB.addEdge(bacon2apollo, bacon, apollo, EdgeKind.BIDIRECTIONAL, 1 /**weight**/); Ingest
- 18. Scaling Writes • Big/Fast data demands write performance • Most NoSQL solutions allow you to scale writes by… – Partitioning the data – Understanding your consistency requirements – Allowing you to defer conflicts Ingest
- 19. App-2 (Ingest V2) App-2 (E23{ V2V3}) Scaling Graph Writes ACID Transactions InfiniteGraph Objectivity/DB Persistence Layer App-1 (Ingest V1) App-3 (Ingest V3) V1 V2 V3 App-1 (E1 2{ V1V2}) App-3 E12 E23 Ingest
- 20. High Performance Edge Ingest IG Core/API C1 C2 C3 E12 E23 TargetContainers PipelineContainers E(1->2) E(3->1) E(2->3) E(2->1) E(2->3) E(3->1) E(1->2) E(3->2) E(1->2) E(2->3) E(3->1) E(2->1) E(2->3) E(3->1) E(3->2) E(1->2) Pipeline Agent Ingest
- 21. Trade offs • Excellent for efficient use of page cache • Able to maintain full database consistency • Achieves highest ingest rate in distributed environments • Almost always has highest “perceived” rate • Trading Off : • Eventual consistency in graph (connections) • Updates are still atomic, isolated and durable but phased • External agent performs graph building Ingest
- 22. Result… 1 client 2 clients 4 clients 8 clients 0 50000 100000 150000 200000 250000 300000 350000 400000 450000 500000 1 2 4 NodesandEdgespersecond 1 client 2 clients 4 clients 8 clients Ingest
- 23. Scaling Reads and Query Distributed API Application(s) Partition 1 Partition 3Partition 2 Partition ...n Processor Processor Processor Processor Partitioning and Read Replicas… easy right !
- 24. Why are Graphs Different ? Distributed API Application(s) Partition 1 Partition 3Partition 2 Partition ...n Processor Processor Processor Processor Navigate
- 25. Distributed Navigation • Detect local hops and perform in memory traversal • Send the partial path to the distributed processing to continue the navigation. • Intelligently cache remote data when accessed frequently • Route tasks to other hosts when it is optimal Navigate
- 26. Distributed Navigation Server Processor Distributed API Partition 1 Partition 2 Processor Application A X Y B C D E P(A,B,C,D) F G Navigate
- 27. GraphViews Leveraging Schema in the Graph Patient Prescription Drug Ingredient Outcome Complaint Visit Allergy Physician Navigate
- 28. Schema Enables Views • GraphViews are extremely powerful • Allow Big Data to appear small ! • Connection inference can lead to exponential gains in query performance • Views are reusable between queries • Views can be persisted • Built into the native kernel Navigate
- 29. Problem of Supernodes In Graph Theory, a “supernode” is a vertex with a disproportionally high number of connected edges. Supernodes make it difficult to do a navigational query in real-time due to the amount of effort it may be to pursue paths through it that may be unfruitful. Navigate Images generated using the IG Visualizer
- 30. Supernodes in Bacon Navigate In the IMDB data set, some examples of supernodes may be talk shows, awards shows, compilations or variety shows. Images generated using the IG Visualizer
- 31. How to avoid supernodes 1. Setting policies on the navigation like the NoRevisitPolicy , MaximumResultCountPolicy and MaximumPathDepthPolicy can be used to customize the overall behavior of the navigation. PolicyChain policies = new PolicyChain(); // Only traverse the same vertex once policies.addPolicy(new NoRevisitPolicy()); // limits the number of paths that will be returned to 10K policies.addPolicy(new MaximumResultCountPolicy(10000)); // limits the path depth to 6 policies.addPolicy(new MaximumPathDepthPolicy(6)); Navigate
- 32. How to avoid supernodes 2. Graph View to exclude or limit types GraphView view = new GraphView(); //Excludes all instances of TvShow from navigation view.excludeClass(myDb.getTypeId(TvShow.class.getName())); //Excludes all movies made for TV/Video view.excludeClass(myDb.getTypeId(Movie.class.getName()), “details.madeForTv || details.madeForVideo”); //Include ActedIn w/ characterName not containing “Himself” view.excludeClass(myDb.getTypeId(WorkedOn.class.getName())); view.includeClass(myDb.getTypeId(ActedIn.class.getName()), “!CONTAINS(characterName, “Himself”)”); Navigate Kevin Bacon Actor The Following TV Show Behind the Scenes Movie Apollo 13 Movie HimselfRyan Hardy Jack Swigert
- 33. How to avoid supernodes 3. Using these policies and graph view, we can filter the size of the result set in our navigation: Navigator navigator = bacon.navigate(view, Guide.SIMPLE_BREADTH_FIRST, Qualifier.ANY, new VertexPredicate(Person.class, ""), policies, myResultHandler); navigator.start(); Navigate
- 34. Filtered Views in Bacon The results of this navigation would look something like this… Navigate Images generated using the IG Visualizer
- 35. Why InfiniteGraph™? • Objectivity/DB is a proven foundation – Building distributed databases since 1993 – A complete database management system • Concurrency, transactions, cache, schema, query, indexing • It’s a Graph Specialist ! – Simple but powerful API tailored for data navigation. – Easy to configure distribution model
- 36. Advanced Configured Placement • Physically co-locate “closely related” data • Driven through a declarative placement model • Dramatically speeds “local” reads Facility Data Page(s)Patient Data Page(s) Mr Citizen Visit Visit Dr Jones San Jose Facility Dr Smith Primary Physician HasHas With At Located Located Facility Data Page(s) Dr Blake Sunny- vale Dr Quinn Located Located With At
- 37. Fully Distributed Data Model Zone 2Zone 1 HostA IG Core/API Distributed Object and Relationship Persistence Layer Customizable Placement HostB HostC HostX AddVertex()
- 38. Polyglot NoSQL Architectures Distributed Data Processing Platform Document Graph Database RDBMS Partitioned Distributed DB (often Document / KV) Users Applications External/LegacyData TransformationMDM Business
- 39. What else! • Distributed update. Update … we are working on it.
- 40. Conclusion I hope that you enjoyed the bacon. My apologies to my kosher friends for any offense. Look out for new features coming soon!
- 41. QUESTIONS?