An Overview of the Emerging Graph Landscape (Oct 2013)
21 likes4,771 views
The document provides an overview of the emerging graph landscape, exploring why graph databases are increasingly relevant for big data analysis and detailing various graph models and use cases across industries. It highlights the evolution of web search, online recruiting, and practical applications in network management as well as core industry sectors like finance and healthcare that benefit from graphs. Additionally, it discusses the characteristics of graph databases and compute engines, along with their integration and deployment in real-world applications.
![Neo Technology, Inc Confidential
“[I]t is arguable that graph databases will have a
bigger impact on the database landscape than
Hadoop or its competitors.”
- Bloor Research, May 2012
http://www.bloorresearch.com/blog/IM-Blog/2012/5/graph-databases-nosql.html
Graph Buzz
6Wednesday, October 2, 13](https://image.slidesharecdn.com/an-overview-of-the-graph-landscape-2013-10-02-slideshare-131003014539-phpapp02/85/An-Overview-of-the-Emerging-Graph-Landscape-Oct-2013-6-320.jpg)
![Neo Technology, Inc Confidential
MATCH (philip:Person)-[:IS_FRIEND_OF]->(friend),
(friend)-[:LIKES]->(restaurant),
(restaurant)-[:LOCATED_IN]->(newyork:Location),
(restaurant)-[:SERVES]->(sushi:Cuisine)
WHERE philip.name = 'Philip' AND newyork.location='New York' AND
sushi.cuisine='Sushi'
RETURN restaurant.name
* Cypher query language examplehttp://maxdemarzi.com/?s=facebook
13Wednesday, October 2, 13](https://image.slidesharecdn.com/an-overview-of-the-graph-landscape-2013-10-02-slideshare-131003014539-phpapp02/85/An-Overview-of-the-Emerging-Graph-Landscape-Oct-2013-13-320.jpg)
![Network Management - Create
CREATE
! (crm {name:"CRM"}),
! (dbvm {name:"Database VM"}),
! (www {name:"Public Website"}),
! (wwwvm {name:"Webserver VM"}),
! (srv1 {name:"Server 1"}),
! (san {name:"SAN"}),
! (srv2 {name:"Server 2"}),
! (crm)-[:DEPENDS_ON]->(dbvm),
! (dbvm)-[:DEPENDS_ON]->(srv2),
! (srv2)-[:DEPENDS_ON]->(san),
! (www)-[:DEPENDS_ON]->(dbvm),
! (www)-[:DEPENDS_ON]->(wwwvm),
! (wwwvm)-[:DEPENDS_ON]->(srv1),
! (srv1)-[:DEPENDS_ON]->(san)
Practical Cypher
17Wednesday, October 2, 13](https://image.slidesharecdn.com/an-overview-of-the-graph-landscape-2013-10-02-slideshare-131003014539-phpapp02/85/An-Overview-of-the-Emerging-Graph-Landscape-Oct-2013-17-320.jpg)
![Network Management - Impact Analysis
// Server 1 Outage
MATCH (n)<-[:DEPENDS_ON*]-(upstream)
WHERE n.name = "Server 1"
RETURN upstream
Practical Cypher
upstream
{name:"Webserver VM"}
{name:"Public Website"}
18Wednesday, October 2, 13](https://image.slidesharecdn.com/an-overview-of-the-graph-landscape-2013-10-02-slideshare-131003014539-phpapp02/85/An-Overview-of-the-Emerging-Graph-Landscape-Oct-2013-18-320.jpg)
![Network Management - Dependency Analysis
// Public website dependencies
MATCH (n)-[:DEPENDS_ON*]->(downstream)
WHERE n.name = "Public Website"
RETURN downstream
Practical Cypher
downstream
{name:"Database VM"}
{name:"Server 2"}
{name:"SAN"}
{name:"Webserver VM"}
{name:"Server 1"}
19Wednesday, October 2, 13](https://image.slidesharecdn.com/an-overview-of-the-graph-landscape-2013-10-02-slideshare-131003014539-phpapp02/85/An-Overview-of-the-Emerging-Graph-Landscape-Oct-2013-19-320.jpg)
![Network Management - Statistics
// Most depended on component
MATCH (n)<-[:DEPENDS_ON*]-(dependent)
RETURN n,
count(DISTINCT dependent)
AS dependents
ORDER BY dependents DESC
LIMIT 1
Practical Cypher
n dependents
{name:"SAN"} 6
20Wednesday, October 2, 13](https://image.slidesharecdn.com/an-overview-of-the-graph-landscape-2013-10-02-slideshare-131003014539-phpapp02/85/An-Overview-of-the-Emerging-Graph-Landscape-Oct-2013-20-320.jpg)
![Neo Technology, Inc Confidential
Overview of Popular
Graph Data Models
• Property Graph
• Description: A “directed, labeled, attributed, multi-
graph”1 which exposes three building blocks: nodes, typed
relationships and key-value properties on both nodes and
relationships
• Vendors: Neo4j, OrientDB, InfiniteGraph, Dex
• RDF Triples
• Description: URI-centered subject-predicate-object
triples as pioneered by the semantic web movement2
• Vendors: AllegroGraph, Sesame
• HyperGraph
• Description: A generalized graph where a
relationship can connect an arbitrary amount of nodes
(compared to the more common binary graph models)3
• Vendors: HyperGraphDB,TrinityDB
1] Rodriguez, M.A., Neubauer, P., “Constructions from Dots and Lines,” 2010, http://arxiv.org/abs/1006.2361
2] W3C,“The Resource Description Framework (RDF),” 2004, http://www.w3.org/RDF/
3] Wikipedia, http://en.wikipedia.org/wiki/Hypergraph
30Wednesday, October 2, 13](https://image.slidesharecdn.com/an-overview-of-the-graph-landscape-2013-10-02-slideshare-131003014539-phpapp02/85/An-Overview-of-the-Emerging-Graph-Landscape-Oct-2013-30-320.jpg)
![Neo Technology, Inc Confidential
1.What is a
Graph Database
A graph database is an online (“real-time”)
database management system with CRUD
methods that expose a graph data model1
• Two important properties:
• Native graph processing, including
index-free adjacency to facilitate traversals
• Native graph storage engine, i.e.
written from the ground up to be
optimized for managing graph data
1] Robinson,Webber, Eifrem. Graph Databases. O’Reilly, 2013. p. 5. ISBN-10: 1449356265
32Wednesday, October 2, 13](https://image.slidesharecdn.com/an-overview-of-the-graph-landscape-2013-10-02-slideshare-131003014539-phpapp02/85/An-Overview-of-the-Emerging-Graph-Landscape-Oct-2013-32-320.jpg)
An Overview of the Emerging Graph Landscape (Oct 2013)
- 1. Neo Technology, Inc Confidential An Overview Of The Emerging Graph Landscape DataWeek Oct 2, 2013 Emil Eifrem emil@neotechnology.com @emileifrem #neo4j 1Wednesday, October 2, 13
- 2. Neo Technology, Inc Confidential Agenda 1. Why Graphs,Why Now? 2. What Is A Graph, Anyway? 3. Graphs In The Real World 4. The Graph Landscape i) Popular Graph Models ii) Graph Databases iii)Graph Compute Engines 2Wednesday, October 2, 13
- 3. Neo Technology, Inc Confidential Why Graphs, Why Now? 3Wednesday, October 2, 13
- 4. Graph Buzz 4Wednesday, October 2, 13
- 5. Neo Technology, Inc Confidential “Graph analysis is the true killer app for Big Data.” - Forrester Research, Dec 2011 http://blogs.forrester.com/james_kobielus/11-12-19-the_year_ahead_in_big_data_big_cool_new_stuff_looms_large Graph Buzz 5Wednesday, October 2, 13
- 6. Neo Technology, Inc Confidential “[I]t is arguable that graph databases will have a bigger impact on the database landscape than Hadoop or its competitors.” - Bloor Research, May 2012 http://www.bloorresearch.com/blog/IM-Blog/2012/5/graph-databases-nosql.html Graph Buzz 6Wednesday, October 2, 13
- 7. Neo Technology, Inc Confidential Graph Buzz Ref: http://www.gartner.com/id=2081316 Copy of Gartner slide: 7Wednesday, October 2, 13
- 8. Neo Technology, Inc Confidential Graph Buzz 8Wednesday, October 2, 13
- 9. Evolution of Web Search Survival of the Fittest Pre-1999 WWW Indexing Discrete Data 1999 - 2012 Google Invents PageRank Connected Data (Simple) 2012-? Google Knowledge Graph, Facebook Graph Search Connected Data (Rich) 9Wednesday, October 2, 13
- 10. Evolution of Online Recruiting 1999 Keyword Search Discrete Data Survival of the Fittest 2011-12 Social Discovery Connected Data 10Wednesday, October 2, 13
- 11. Neo Technology, Inc Confidential What Is A Graph, Anyway? 11Wednesday, October 2, 13
- 12. Neo Technology, Inc Confidential 12Wednesday, October 2, 13
- 13. Neo Technology, Inc Confidential MATCH (philip:Person)-[:IS_FRIEND_OF]->(friend), (friend)-[:LIKES]->(restaurant), (restaurant)-[:LOCATED_IN]->(newyork:Location), (restaurant)-[:SERVES]->(sushi:Cuisine) WHERE philip.name = 'Philip' AND newyork.location='New York' AND sushi.cuisine='Sushi' RETURN restaurant.name * Cypher query language examplehttp://maxdemarzi.com/?s=facebook 13Wednesday, October 2, 13
- 14. Neo Technology, Inc Confidential 14Wednesday, October 2, 13
- 15. Neo Technology, Inc Confidential What drugs will bind to protein X and not interact with drugY? Of course.. a graph is a graph is a graph 15Wednesday, October 2, 13
- 17. Network Management - Create CREATE ! (crm {name:"CRM"}), ! (dbvm {name:"Database VM"}), ! (www {name:"Public Website"}), ! (wwwvm {name:"Webserver VM"}), ! (srv1 {name:"Server 1"}), ! (san {name:"SAN"}), ! (srv2 {name:"Server 2"}), ! (crm)-[:DEPENDS_ON]->(dbvm), ! (dbvm)-[:DEPENDS_ON]->(srv2), ! (srv2)-[:DEPENDS_ON]->(san), ! (www)-[:DEPENDS_ON]->(dbvm), ! (www)-[:DEPENDS_ON]->(wwwvm), ! (wwwvm)-[:DEPENDS_ON]->(srv1), ! (srv1)-[:DEPENDS_ON]->(san) Practical Cypher 17Wednesday, October 2, 13
- 18. Network Management - Impact Analysis // Server 1 Outage MATCH (n)<-[:DEPENDS_ON*]-(upstream) WHERE n.name = "Server 1" RETURN upstream Practical Cypher upstream {name:"Webserver VM"} {name:"Public Website"} 18Wednesday, October 2, 13
- 19. Network Management - Dependency Analysis // Public website dependencies MATCH (n)-[:DEPENDS_ON*]->(downstream) WHERE n.name = "Public Website" RETURN downstream Practical Cypher downstream {name:"Database VM"} {name:"Server 2"} {name:"SAN"} {name:"Webserver VM"} {name:"Server 1"} 19Wednesday, October 2, 13
- 20. Network Management - Statistics // Most depended on component MATCH (n)<-[:DEPENDS_ON*]-(dependent) RETURN n, count(DISTINCT dependent) AS dependents ORDER BY dependents DESC LIMIT 1 Practical Cypher n dependents {name:"SAN"} 6 20Wednesday, October 2, 13
- 21. Neo Technology, Inc Confidential Graphs In The Real World 21Wednesday, October 2, 13
- 22. Neo Technology, Inc Confidential Core Industries & Use Cases: Web / ISV Finance & Insurance Telecomm- unications Network & Data Center Management MDM Social Geo Early Adopter Segments (What we expected to happen - view from several years ago) 22Wednesday, October 2, 13
- 23. Neo Technology, Inc Confidential Core Industries & Use Cases: Web / ISV Finance & Insurance Telecomm- unications Network & Data Center Management MDM Social Geo Select Commercial Customers* Across Anticipated Segments Neo4j Adoption Snapshot Core Industries & Use Cases: Software Financial Services Telecomm unications Health Care & Life Sciences Web Social, HR & Recruiting Media & Publishing Energy, Services, Automotive, Gov’t, Logistics, Education, Gaming, Other Network & Data Center Management MDM / System of Record Social Geo Recommend- ations Identity & Access Mgmt Content Management BI, CRM, Impact Analysis, Fraud Detection, Resource Optimization, etc. Accenture Finance Energy Aerospace 23Wednesday, October 2, 13
- 24. Neo Technology, Inc Confidential • Network Graph (e.g. Network Dependency Analysis, Network Inventory, etc.) • Social Graph (mobile apps, social recommendations, collaboration) • Call Graph (creating inferred social graph, churn reduction, etc.) • Master Data Graph (org & product hierarchy, data governance, IAM) • Help Desk Graph (enterprise collaboration) 5 Graphs of Telco 24Wednesday, October 2, 13
- 25. Neo Technology, Inc Confidential • Payment Graph (e.g. Fraud Detection, Credit Risk Analysis, Chargebacks...) • Customer Graph (org drillthru, product recommendations, mobile payments, etc.) • Entitlement Graph (identity & access management, authorization) • Portfolio Graph (portfolio analytics, risk analysis, trading, compliance) • Master Data Graph (enterprise collaboration, corporate hierarchy, data governance) 5 Graphs of Finance Finance 25Wednesday, October 2, 13
- 26. Neo Technology, Inc Confidential • Provider Graph (e.g. referrals, patient management, research) • Patient Graph (support communities, doctor recommendations, clinical trials) • Bioinformatic Graph (drug research, genetic screening, plant engineering, etc.) • Master Data Graph (biological master data, evolutionary taxonomy, etc.) • Treatment Graph (collaborative medicine, clinical trials, etc.) 5 Graphs of Health Care 26Wednesday, October 2, 13
- 27. Accenture Background •One of the world’s largest logistics carriers •Projected to outgrow capacity of old system •New parcel routing system •Single source of truth for entire network •B2C & B2B parcel tracking •Real-time routing: up to 5M parcels per day Business problem •24x7 availability, year round •Peak loads of 2500+ parcels per second •Complex and diverse software stack •Need predictable performance & linear scalability •Daily changes to logistics network: route from any point, to any point Solution & Benefits •Neo4j provides the ideal domain fit: •a logistics network is a graph •Extreme availability & performance with Neo4j clustering •Hugely simplified queries, vs. relational for complex routing •Flexible data model can reflect real-world data variance much better than relational •“Whiteboard friendly” model easy to understand Industry: Logistics Use case: Parcel Routing Neo Technology Confidential 27Wednesday, October 2, 13
- 28. Neo Technology, Inc Confidential Industry: Online Job Search Use case: Social / Recommendations • Online jobs and career community, providing anonymized inside information to job seekers Business problem • Wanted to leverage known fact that most jobs are found through personal & professional connections • Needed to rely on an existing source of social network data. Facebook was the ideal choice. • End users needed to get instant gratification • Aiming to have the best job search service, in a very competitive market Solution & Benefits • First-to-market with a product that let users find jobs through their network of Facebook friends • Job recommendations served real-time from Neo4j • Individual Facebook graphs imported real-time into Neo4j • Glassdoor now stores > 50% of the entire Facebook social graph • Neo4j cluster has grown seamlessly, with new instances being brought online as graph size and load have increased Person Company KNOW S Person Person KNOWS Company KNOWS WORKS_AT WORKS_AT Background Sausalito, CA 28Wednesday, October 2, 13
- 29. Neo Technology, Inc Confidential The Graph Landscape 29Wednesday, October 2, 13
- 30. Neo Technology, Inc Confidential Overview of Popular Graph Data Models • Property Graph • Description: A “directed, labeled, attributed, multi- graph”1 which exposes three building blocks: nodes, typed relationships and key-value properties on both nodes and relationships • Vendors: Neo4j, OrientDB, InfiniteGraph, Dex • RDF Triples • Description: URI-centered subject-predicate-object triples as pioneered by the semantic web movement2 • Vendors: AllegroGraph, Sesame • HyperGraph • Description: A generalized graph where a relationship can connect an arbitrary amount of nodes (compared to the more common binary graph models)3 • Vendors: HyperGraphDB,TrinityDB 1] Rodriguez, M.A., Neubauer, P., “Constructions from Dots and Lines,” 2010, http://arxiv.org/abs/1006.2361 2] W3C,“The Resource Description Framework (RDF),” 2004, http://www.w3.org/RDF/ 3] Wikipedia, http://en.wikipedia.org/wiki/Hypergraph 30Wednesday, October 2, 13
- 31. Neo Technology, Inc Confidential Graph Ecosystem @ 10k Feet 1. Graph Databases 2. Graph Compute Engines 31Wednesday, October 2, 13
- 32. Neo Technology, Inc Confidential 1.What is a Graph Database A graph database is an online (“real-time”) database management system with CRUD methods that expose a graph data model1 • Two important properties: • Native graph processing, including index-free adjacency to facilitate traversals • Native graph storage engine, i.e. written from the ground up to be optimized for managing graph data 1] Robinson,Webber, Eifrem. Graph Databases. O’Reilly, 2013. p. 5. ISBN-10: 1449356265 32Wednesday, October 2, 13
- 33. Neo Technology, Inc Confidential Graph Local Queries e.g. Recommendations, Friend-of-Friend, Shortest Path Sweet Spot for Graph Databases 33Wednesday, October 2, 13
- 34. Neo Technology, Inc Confidential The Emerging Graph Database Space Graph Storage GraphProcessing N on- N ative Native Native FlockDB AllegroGraph The Graph Database Space 34Wednesday, October 2, 13
- 35. Neo Technology, Inc Confidential Processing platforms that enable graph global computational algorithms to be run against large data sets Graph Compute Engine (Working Storage) In-Memory Processing System(s) of Record Graph Compute Engine Data extraction, transformation, and load 2.What is a Graph Compute Engine 35Wednesday, October 2, 13
- 36. Neo Technology, Inc Confidential How many restaurants, on average, has each person liked? Graph Global Queries Sweet Spot for Graph Compute Engines 36Wednesday, October 2, 13
- 37. Neo Technology, Inc Confidential Graph Compute Engines • In-Memory / Single Machine • Distributed - most common of which is the “Bulk Synchronous Parallel Model” (aka Pregel clone) Largely fall into one of two patterns: 37Wednesday, October 2, 13
- 38. Neo Technology, Inc Confidential Distributed Computing Architecture - Examples Graph Compute Engine • Apache project based on Hadoop • Bulk Synchronous Processing Model (Pregel Clone) • Released in 2012 • OSS Project developed out of CMU • Based on Hadoop & Map/Reduce • Includes key algos for graph global pattern matching & visualization • OSS Project • Distributes relationships vs. nodes • Developed at CMU with funding from DARPA, Intel, et al. &VC 38Wednesday, October 2, 13
- 39. Neo Technology, Inc Confidential Cassovary • OSS Project led by Twitter • Used by Twitter for large- scale graph mining (uses daily export from FlockDB system of record) • “Not designed for persistence or database functionality”. YarcData uRiKA • Graph compute appliance launched by Cray in Feb 2012 • Build to discover unforeseen relationships in the graph In-Memory Single-Machine Examples Graph Compute Engine GraphChi • GraphLab Spinoff • Similar order-of-magnitude performance as GraphLab on a Mac Mini 39Wednesday, October 2, 13
- 40. Neo Technology, Inc Confidential Example Graph Database Deployment Application Other Databases ETL Graph Database Cluster Data Storage & Business Rules Execution Reporting Graph- Dashboards & Ad-hoc Analysis Graph Visualization End User Ad-hoc visual navigation & discovery Bulk Analytic Infrastructure (e.g. Graph Compute Engine) ETL Graph Mining & Aggregation Data Scientist Ad-Hoc Analysis 40Wednesday, October 2, 13
- 41. Neo Technology, Inc Confidential DEAR DATA SCIENTIST: TAKE THE RED PILL JOIN THE GRAPH. WE ARE HIRING. 41Wednesday, October 2, 13
- 42. Neo Technology, Inc Confidential teh end (sic) stay connected 42Wednesday, October 2, 13