Powering a Graph Data System with Scylla + JanusGraph

Editor's Notes

• #2: Let's give another round of applause to Brian.  Everything he said applies here – now we'll just dig into the technical pieces a bit more.
• #3: I'm Ryan Stauffer, I'm the founder and CEO of a Bay Area startup called Enharmonic.  I first got excited about graph databases several years back when I was leading data analytics and strategy for a large automotive aftermarket company.  We were trying to build a unified model of data for the automotive aftermarket that combined data from across our different verticals.  Using the source data in its existing form – hundreds of tables, and hundreds of millions of rows & columns - was leading us down a really bad path.  It became clear that insights would be much easier if we used a graph data model, where we can explicitly model our data as real-world business concepts.  Ever since then, I’ve viewed graph data systems as a core part of the solution for how to ask and answer better questions about our businesses.
• #4: For a litle backdrop about what we'll be talking about – what do we do at Enharmonic?  Well, we're working to solve the problem of how companies interact with their data. We provide a clean, visual interface that let's business decision makers directly access their data with free-text search and point-click-and-drag actions.  Data is modeled and retrieved as logical business concepts like Customers, Products, and Orders.  Our system recommends analyses that make sense based on the data, and then goes ahead and executes those with just a few clicks.  To make this possible, we use lots of automation on the backend – and sitting behind everything, we use a graph data system.
• #5: Brian discussed graphs in the last session, so I'm not going to rehash everything, but I do want to do a brief level-set.  So what do I mean when I say "Graph Data System"?
• #6: We can break that into 2 parts: "Graph" & "Data System" By "graph" we mean that we're modelling our data as a property graph, using Vertices, Edges & Properties. Vertices model entities like Customers or Products Edges model relationships between entities, like how one Customer KNOWS another Customer, or a Customer HAS PURCHASED a Product. Properties model attributesof entites and relationships, like the name and age of a Customer. By "Data System" we mean that several distinct components combine to form a single, logical system.
• #7: There are several options for graph databases out there on the market, but when we need a combination of scalability, flexibility, and performance, we can look to a system built of JanusGraph, Scylla, and Elasticsearch. This is a single logical data system is structured into 3 parts: - In the center we have JanusGraph, a Java application that clients communicate with directly. - It serves as the abstraction layer that let's us interact with our data as a graph. - JG will write to and read from Scylla, where our data is ultimately persisted. - We can optionally add Elasticsearch to help us with advanced indexing and text search capabilities
• #8: So that sounds interestnig, but why do we want to do this at all?
• #9: I think there are 3 core benefits of this graph data system. - Flexibility - Schema support - Support for both transactional & analytical workloads
• #10: The killer feature of using a graph is its flexibility - Business logic changes, application requirements change, and it can often be a real problem trying to support that with traditional databases - Using a graph means our data model isn't set in stone. - We can iterate and evolve the data model by adding additional vertices and edges to meet our new needs, without throwing out everything that already works. - We can also write analytics results directly back to the graph, explicitly connecting to our primary data. - This simplifies the ways that teams can collaborate and share insights, while allowing for powerful data provenance capabilities.
• #11: Schema support is a real "nice-to-have" when it comes to separating business logic from lower-level database integrity issues. JanusGraph, unlike some other graph databases out there, supports defining a schema for data, but doesn't require that we do this. Basically, we can apply useful constraints to what is allowed and disallowed on our graph. For example, we can ensure that name and age properties are only allowed to be written to a Customer vertex, but we don't required that every Customer vertex have all of these properties (minimizes the need for pointless null field values!) We can also specify that a Product and Customer vertex are allowed to be related with a HAS_PURCHASED edge, bu we don't required that each Product vertex must have that edge. This sort of clear schema flexibility is difficult to replicate outside of a graph environment. Separates data integrity mantenance from our business logic – letting our DB take over DB tasks, without offloading them onto the application layer.
• #12: - Finally, with this graph data system, we can execute both transactional and analytical workloads with the same data systtem and same query language – Gremlin. - We access data by “traversing” our graph, travelling from vertex to vertex by means of connecting edges. - We can think of a transactional workload to be one where we travel to a small number of vertices and edge, and where our goal is to minimize latency. - An analytical workload, on the other hand, is one where we travel to all, or a substantial portion, of our vertices and edges.  Our goal here is to maximize throughput. - Backed by the high-IO performance of Scylla, we can achieve scalable, single-digit millisecond response for transactional workloads.  We can also leverage Spark to handle large scale analytical workloads
• #13: It's easy to talk about all of this in theory, but how do we go about actually deploying it
• #14: 1st of all, WHERE are we going to deploy this? In a production environment, it makes sense to deploy Scylla on either VMs or bare metal. For JanusGraph & ES, there are many advantages to deploying on Kubernetes Q – Quick show of hands, who is using Kubernetes today? Q – Who has tried deploying Scylla on top of Kubernetes? (Yannis Zarkadas gave a great talk earlier today on using the Scylla Operator to manage Scylla on K8s – if you missed it I highly recommend checking out the talk online.)
• #15: Kubernetes is an open source system for managing containerized applications Allows you to group and manage application containers as logical units Fundamentally, its about building and interacting with abstractions on top of basic resources (Compute, memory, disk, network) Not going to touch every last detail of the k8s manifests, but I want really dive into the low-level fundamental of the k8s resources you'll be using. Now even when setting up our pieces on k8s seems pedantic, remember that this greatly simplifies the process of installing and managing a complex application.  As many of you probably know, it's significantly easier to do it this way versus installing and upgrading each app and their dependencies manually at the VM level.
• #16: Walkthrough the details of deploying the whole system. Big picture, we have 2 types of components – stateful and stateless Stateful components are Scylla and Elasticsearch, where we'll actually persists our data.  Everything else is stateless and ephemeral.  Our actual JanusGraph app pods for instance are stateless, and if one dies, we simply spin up a new one in its place. The what does this looks like? A client (maybe an app, maybe our little Scylla monster up here) and she'll issue queries to JanusGraph. - Those queries hit a load balancer and are passed to 1 or more pods managed as part of a JanusGraph deployment. JanusGraph app is what presents the "graph" view of data, and it does it by intermediating between the client and stateful apps. Most data is put in Scylla, over here on the left. For more advanced indexing, we use Elasticsearch, which we deploy as a Stateful Set and Headless Service.
• #17: Diving into more detail, we start with Scylla. We can actually use your existing Scylla cluster, meaning there's 0 lift! The one thing we'll do is create a new keyspace to hold graph data.
• #18: To give us more advanced indexing capabilities, we'll deploy Elasticsearch as well. We deploy it on Kubernetes in 3 parts. - Headless Service - Stateful Set - Storage Class ES is stateful, so needs to persist data, which we'll accomplish this by means of a stateful set. Now, a stateful set is just used to manage 1 or more replica pods, which are the nodes in our ES cluster.  But it does this in a unique way.  It assigns numbers to each pod and the disks that are mounted to it.  This way, we consistently mount the same disk to the same pod #. This gives us a reliably stateful system, where even if individual pods fail, they're safely recreated automatically by Kubernetes.
• #19: We define a storage class – what type of disks do we want to mount to our Elasticsearch nodes?  In this case, we'll choose SSDs. We'll define a headless service.  We set clusterIP to None, specify our standard ES ports, and provide a selctor to target our stateful set pods. The last step is to define our stateful set.  This references the Storage Class and Headless Service we just defined, so I color-coded the important bits. For storage, shown in blue, our goal is to define a disk from our elasticsearch-ssd storage class for each ES node, and mount it to that node.  To do this, we'll define a Volume Clam Template, and define a volume mount that mounts the disk at our ES data path. For networking, shown in red, we specify the Headless Service name.  We'll also define 1 environment variable, that allows for ES node discovery. Q – I THINK THERE'S A TYPO HERE ON THE SELECTOR FOR THE HEADLESS SERVICE.
• #20: Assuming we put all of this into a single manifest file, we can deploy Elasticsearch to our Kubernetes cluster with a single "apply" command After a little bit of initialization, we can see the Ready status of our stateful set, the 3 pods it controls, and the services that routes network traffic to these pods.
• #21: Now, for the last and most important piece of the puzzle – JanusGraph. We'll deploy this on Kubernetes as well.
• #22: There are already official JanusGraph images available on Docker Hub, and for these examples we'll be using version 0.4.0 You could also build your own using the JanusGraph project build scripts, and push that image to a private image repository (for example, Google Cloud Platform)
• #23: Now how do we use JanusGraph?  Let's start with a minimal example.  Not for production use - but illustrates how this all works. We'll deploy a single pod to get console access to our system.
• #24: We'll run JanusGraph in a single pod, and connect to it directly. That means that the graph is only accessible through the console connection, but all of our actions are still persisted in Scylla and Elasticsearch. Now, the standard JanusGraph docker image includes some great templateing and presets, which allow us to configure out connection to our storage and indexing backends with just a few environment variables. We're using Scylla * Elasticsearch, so we set cql-es as our JanusGraph properties template. We set the hostname as 1 or more of the Scylla cluster hostnames We set the keyspace as a new, clean Scylla keyspace where we'll store all of our graph data. Finally, supply the K8s cluster hostnames for our Elasticsearch nodes.
• #25: With that manifest file, we can create a pod, then connect to it with an interactive terminal. This will bring up a Gremlin Console. The JG Docker image will prepopuate a standard janusgraph.properties file that will reflect the env var configuration we just setup. We use a factory to create a graph instance, and then we can do whatever we'd like to! For example, we can start by defining a schema for a Product vertex with name and productId properties.
• #26: If we want to actually move to a real environment, we need to support multiple users and applications, probably written in different languages. To handle this we deploy JanusGraph server. On Kubernetes, we'll do this as a Deployment, which manages 1 or more stateless replica pods. We put a load balancer in front of it, exposed on an external or internal IP depending on the use case.
• #27: When we deploy JanusGraph as a standalone server, we're actually using the Apache TinkerPop Gremlin Server underneath the hood, which will accept Gremlin language queries issued from applications written in multiple languages (Python, Java, JS, etc.) The Service is pretty simple just a LoadBalancer that will route network requests to our pods.  We're using port 8182 because that's the standard gremlin websocket port. We manage those pods as a single deployment.  We specify the number of replicas, the image, and setup the environment variables just like we did before.
• #28: We apply our manifest, and check that everything is running.  The key parts are the Load Balancer and Deployment. Once our LB has its IP assigned, we're able to connect to our JG pods with a client application.  Now we can issue queries, store data – do whatever we want! Now, some of that description of K8s manifest got pretty pedantic.  There's got to be a better way, right?
• #29: There is – Helm Charts! Q – With a show of hands, who uses Helm Charts? Awesome.  We can think of Helm as a package manager for k8s.  It lets us template out and group related manifest files into logical packages called Charts. This makes it easy to define, deploy and upgrade Kubernetes applications with single commands. We just released our own opinionate take on how to deploy JanusGraph as a Helm Chart on Github.  If you like saving time and energy, please check it out and use it
• #30: Kubernetes gives us tremendous power, and makes it easy to deploy JanusGraph on top of Scylla.
• #31: With our deployment up and running, we have a flexible, scalable graph data system that we can use as the bedrock for an exciting new generation of applications.
• #32: Thank you for your time. If you'd like to stay in touch, you can follow me on Twitter or connect with me on LinkedIn.  You can also contact me directly via email. I think we have a few more minutes, so what questions do you have?