InfiniteGraph
Download as PPT, PDF1 like837 views
InfiniteGraph is a distributed graph database built on Objectivity/DB that provides a simple graph-focused API, automated distribution and deployment, and consistency models that allow for relaxed consistency and high ingest rates of billions of relationships per day. It uses a distributed architecture with partitioning and pipelining to enable navigation across large graphs and scales graph algorithms in a distributed manner by breaking up processes and orchestrating them across partitions.
![Thank you ! Copyright © InfiniteGraph [email_address] Twitter - @infinitegraph](https://image.slidesharecdn.com/infinitegraph-overview-110417143127-phpapp01/85/InfiniteGraph-18-320.jpg)
InfiniteGraph
- 1. An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph
- 2. For today’s discussion… InfiniteGraph Background / History Architecture What we are working on Consistency Models Challenges/Requirements of Distibution Data model – Schema vs Schema-less Copyright © InfiniteGraph
- 3. Building on Objectivity/DB Derived from Objectivity/DB core Distributed Object Database Native core (broad platform support) with C++, Java, C# and Python Bindings Thousands of deployments, many are 24x7x365 Markets : VLDB, Data Fusion / Metadata, Complex Object Models, Relationship Analytics Copyright © InfiniteGraph
- 4. Relationship Analytics Experience Relationship Analytics Large part of Objy/DB Government business Significant growth in the commercial space Many OEM opportunities Mostly based on graph theory Hand coded algorithms Copyright © InfiniteGraph
- 5. Needed something more… Objectivity Provides Powerful distributed object storage Flexible class based persistence Enterprise DB features But…. Lots of custom coding Complex deployment No generic algorithm framework Copyright © InfiniteGraph
- 6. Enter InfiniteGraph Simple Graph focused API Automated distribution and deployment Mostly configuration driven Java class based persistence Property model support Asynchronous navigation Indexing framework Copyright © InfiniteGraph
- 7. Some code… Copyright © InfiniteGraph Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”)); alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob); bob.addEdge(new Call(timestamp), carlos); carlos.addEdge(new Payment(100000.00), charlie); bob.addEdge(new Call(timestamp), charlie); Alice Carlos Charlie Bob Meets Calls Pays Calls
- 8. Basic Architecture Copyright © InfiniteGraph IG Core/API Configuration Navigation Execution Management Extensions Blueprints User Apps Objectivity/DB Distributed Database Session / TX Management Placement
- 10. Targeting Large Graphs Graphs grow quickly Billions of phone calls / day in US Emails, social media events, IP Traffic Financial transactions Some analytics require navigation of large sections of the graph Must distribute data and go parallel Copyright © InfiniteGraph
- 11. Consistency Models Trading off full consistency for performance Relax locking, allow stale reads Gain 100x edge ingest rates Great for “social graph” applications These tend to have relaxed consistency requirements API allows choice per operation Copyright © InfiniteGraph
- 12. Pipelining Copyright © InfiniteGraph IG Core/API Configuration Navigation Execution Management Extensions Session / TX Management Placement (MDP) Placement (Pipelining) V 1 V 2 V 3 E 12 E 23 Pipeline Manager Staging Containers Pipeline Containers E(1->2) E(3->1) E(2->3) E(2->1) E(2->3) E(3->1) E(1->2) E(3->2) E(1->2) E(2->3) E(3->1) E(2->1) E(2->3) E(3->1) E(3->2) E(1->2)
- 13. Distributing Navigation Graph algorithms naturally branch Breaking up the process is relatively simple Orchestrating it is more challenging Copyright © InfiniteGraph Alice Carlos Charlie Bob Meets Calls Pays Dave Eve Chuck Calls Lives With Meets
- 14. Partitioned Graphs are Ugly Copyright © InfiniteGraph Distributed API Application(s) Partition 1 Partition 3 Partition 2 Partition ... n Processor Processor Processor Processor
- 15. Making it scale… A distributed data tier supports parallel IO A distributed cache/processor tier for processing “navlets” Ability to deal with remote data reads (fast) Navlets hit processor/cache instances based on consistent hashing of their start vertex Copyright © InfiniteGraph
- 16. Flexible Data Models Schema isn’t for everyone Great for relatively static models Support for inheritance helps So does auto-evolution Looking to support document style elements Support for ad-hoc attributes Ability to mix both approaches Copyright © InfiniteGraph
- 17. Other Projects Blueprints / Gremlin support Native Visualizer and Management Console Built in Qualifiers More Indexing Enhancements Algorithms Large scale samples Copyright © InfiniteGraph
- 18. Thank you ! Copyright © InfiniteGraph [email_address] Twitter - @infinitegraph