SlideShare a Scribd company logo
Using Spark for
Timeseries Graph
Analytics
SIGMOID 12
TH
MEETUP
-Ved Mulkalwar
ved.mulkalwar@gmail.com
9564211606
Content
 Introduction to sample problem statement
 Which Graph database is used and why
 Installing Titan
 Titan with Cassandra
 The Gremlin Cassandra script: A way to store data in cassandra from Titan
Gremlin
 Accessing Titan with Spark
Introducing which kind of time-series
problems that can be solved using
graph analytics and introducing the
problem statement which I have been
working on solving.
The dynamics of a complex system is usually
recorded in the form of time series. In recent
years, the visibility graph algorithm and the
horizontal visibility graph algorithm have been
recently introduced as the mapping between
time series and complex networks.
Transforming time series into the graphs, the
algorithms allows applying the methods of
graph theoretical tools for characterizing time
series, opening the possibility of building fruitful
connections between time series analysis,
nonlinear dynamics, and graph theory.
 The problem statement which I have been working on:
 Our initial goal was finding anomalies in the given timeseries. We started
with slicing the data into small meaningfull parts so that the data becomes
smaller and contiguous data points having the same property get clubbed
together.
 Later we created a graph with these parts and tried to find the parts having
similar properties. Doing this will single out anomalies which was the goal.
Which graph database I used and why
i.e. Difference between graphX, neo4j
and titan , why we used titan
Titan vs graphX
 The fundamental difference between Titan and GraphX lies in how they
persistence data and how they process data, Titan by default persists data
(vertices and edges and properties) to a distributed data store in the form
of tables bound to a specific schema. This schema can be stored in
Berkley DB tables, Cassandra tables or Hbase tables.
 In the case of Titan the graph is stored as vertices in a vertex table and
edges in an edge table.
 GraphX has no real persistence layer (yet?), yes it can persist to HDFS
files, but it cannot persist to a distributed datastore in a common schema
like form as in Titan’s case.
 A graph only exists in GraphX when it is loaded into memory off raw data
and interpreted as Graph RDDs, Titan stores the graph permanently.
 The other major difference between the two solutions is how they process
graph data.
 By default, GraphX solves queries via distributed processing on many
nodes in parallel where possible as opposed to Titan processing pipelines
on a single node. Titan can also take advantage of parallel processing via
Faunus/HDFS if necessary.
Neo4j vs titan
 The primary difference between Titan and Neo4j is scalability: Titan can
distribute the graph across multiple machines (using either Cassandra or
Hbase as the storage backend) which does three things:
 1) It allows Titan to store really, really large graphs by scaling out
 2) It allows Titan to handle massive write loads
 3) No single point of failure for very high availability. While Neo4j's HA
setup gives you redundancy through replication, death of the master in
such a setup leads to temporary service interruption while a new master is
elected.
 This allows Titan to scale to large deployments with thousands of
concurrent users as demonstrated in the benchmark:
 http://thinkaurelius.com/2012/08/06/titan-provides-real-time-big-graph-data/
 Neo4j has been around much longer and is therefore more "mature" in
some regards:
 1) Integrated with an external lucene index gives it more sophisticated
search capabilities
 2) More integration points into other programming languages and
development frameworks (e.g. spring)
Intro to Titan graph db and how to use
it with cassandra & spark
Installing Titan
 Downloaded the latest prebuilt version (0.9.0-M2) of Titan at
s3.thinkaurelius.com/downloads/titan/titan-0.9.0-M2-hadoop1.zip.
 Carry out the following steps to ensure that Titan is installed on each node
in the cluster:
 Now, use the Linux su (switch user) command to change to the root
account, and move the install to the /usr/local/ location. Change the file
and group membership of the install to the hadoop user, and create a
symbolic link called titan so that the current Titan release can be referred to
as the simplified path called /usr/local/titan:
Titan with Cassandra
 In this section, the Cassandra NoSQL database will be used as a storage
mechanism for Titan. Although it does not use Hadoop, it is a large-scale,
cluster-based database in its own right, and can scale to very large cluster
sizes. A graph will be created, and stored in Cassandra using the Titan
Gremlin shell. It will then be checked using Gremlin, and the stored data
will be checked in Cassandra. The raw Titan Cassandra graph-based data
will then be accessed from Spark. The first step then will be to install
Cassandra on each node in the cluster.
 Install Cassandra on all the nodes
 Set up the Cassandra configuration under /etc/cassandra/conf by altering
the cassandra.yaml file:
 Install Cassandra on all the nodes
 Set up the Cassandra configuration under /etc/cassandra/conf by altering
the cassandra.yaml file:
 Log files can be found under /var/log/cassandra, and the data is stored
under /var/lib/cassandra. The nodetool command can be used on any
Cassandra node to check the status of the Cassandra cluster:
 The Cassandra CQL shell command called cqlsh can be used to access
the cluster, and create objects. The shell is invoked next, and it shows that
Cassandra version 2.0.13 is installed:
 The Cassandra query language next shows a key space called keyspace1
that is being created and used via the CQL shell:
The Gremlin Cassandra script
 The interactive Titan Gremlin shell can be found within the bin directory of
the Titan install, as shown here. Once started, it offers a Gremlin prompt:
 The following script will be entered using the Gremlin shell. The first
section of the script defines the configuration in terms of the storage
(Cassandra), the port number, and the keyspace name that is to be used:
 Next define the generic vertex properties' name and age for the graph to
be created using the Management System. It then commits the
management system changes:
 Now, six vertices are added to the graph. Each one is given a numeric
label to represent its identity. Each vertex is given an age and name value:
 Finally, the graph edges are added to join the vertices together. Each edge
has a relationship value. Once created, the changes are committed to
store them to Titan, and therefore Cassandra:
 This results in a simple person-based graph, shown in the following figure
 This graph can then be tested in Titan via the Gremlin shell using a similar
script to the previous one. Just enter the following script at the gremlin>
prompt, as was shown previously. It uses the same initial six lines to create
the titanGraph configuration, but it then creates a graph traversal variable
g.
 The graph traversal variable can be used to check the graph contents.
Using the ValueMap option, it is possible to search for the graph nodes
called Mike and Flo. They have been successfully found here:
 Using the Cassandra CQL shell, and the Titan keyspace, it can be seen
that a number of Titan tables have been created in Cassandra:
 It can also be seen that the data exists in the edgestore table within
Cassandra:
 This assures us that a Titan graph has been created in the Gremlin shell,
and is stored in Cassandra. Now, I will try to access the data from Spark.
Accessing Titan with Spark
 So far, Titan 0.9.0-M2 has been installed, and the graphs have
successfully been created using Cassandra as backend storage options.
These graphs have been created using Gremlin-based scripts. In this
section, a properties file will be used via a Gremlin script to process a
Titan-based graph using Apache Spark. Cassandra will be used with Titan
as backend storage option.
The following figure, shows the
architecture used in this section.
 Let us examine a properties file that can be used to connect to Cassandra
as a storage backend for Titan. It contains sections for Cassandra, Apache
Spark, and the Hadoop Gremlin configuration. My Cassandra properties
file is called cassandra.properties, and it looks like this
Into Titan code
 The following necessary TinkerPop and Aurelius classes that will be used:
Scala code
Output:
Thank You

More Related Content

PPTX
Using spark for timeseries graph analytics
PPTX
Omid: A transactional Framework for HBase
PPTX
C*ollege Credit: CEP Distribtued Processing on Cassandra with Storm
PDF
Strava Labs: Exploring a Billion Activity Dataset from Athletes with Apache S...
PDF
Predictive Maintenance at the Dutch Railways with Ivo Everts
PDF
Spark Streaming into context
PDF
Time series data monitoring at 99acres.com
PDF
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
Using spark for timeseries graph analytics
Omid: A transactional Framework for HBase
C*ollege Credit: CEP Distribtued Processing on Cassandra with Storm
Strava Labs: Exploring a Billion Activity Dataset from Athletes with Apache S...
Predictive Maintenance at the Dutch Railways with Ivo Everts
Spark Streaming into context
Time series data monitoring at 99acres.com
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...

What's hot (20)

PDF
So you think you can stream.pptx
PPTX
Guest Lecture on Spark Streaming in Stanford CME 323: Distributed Algorithms ...
PDF
Accumulo Summit 2016: GeoMesa: Using Accumulo for Optimized Spatio-Temporal P...
PDF
Apache Spark for Library Developers with William Benton and Erik Erlandson
PDF
Spark streaming: Best Practices
PDF
Monitoring pg with_graphite_grafana
PDF
Time Series Processing with Apache Spark
PPTX
Real Time Data Processing Using Spark Streaming
DOCX
Cloudyn - Multi vendor Cloud management
PPTX
An Introduction to Spark
PDF
Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark
PDF
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
PDF
Ted Dunning – Very High Bandwidth Time Series Database Implementation - NoSQL...
PPTX
Chris Hillman – Beyond Mapreduce Scientific Data Processing in Real-time
PDF
Kafka 102: Streams and Tables All the Way Down | Kafka Summit San Francisco 2019
PDF
Spark
PPTX
Discretized Stream - Fault-Tolerant Streaming Computation at Scale - SOSP
PPTX
High Resolution Energy Modeling that Scales with Apache Spark 2.0 Spark Summi...
PDF
Productionizing your Streaming Jobs
PPTX
Learning spark ch05 - Loading and Saving Your Data
So you think you can stream.pptx
Guest Lecture on Spark Streaming in Stanford CME 323: Distributed Algorithms ...
Accumulo Summit 2016: GeoMesa: Using Accumulo for Optimized Spatio-Temporal P...
Apache Spark for Library Developers with William Benton and Erik Erlandson
Spark streaming: Best Practices
Monitoring pg with_graphite_grafana
Time Series Processing with Apache Spark
Real Time Data Processing Using Spark Streaming
Cloudyn - Multi vendor Cloud management
An Introduction to Spark
Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Ted Dunning – Very High Bandwidth Time Series Database Implementation - NoSQL...
Chris Hillman – Beyond Mapreduce Scientific Data Processing in Real-time
Kafka 102: Streams and Tables All the Way Down | Kafka Summit San Francisco 2019
Spark
Discretized Stream - Fault-Tolerant Streaming Computation at Scale - SOSP
High Resolution Energy Modeling that Scales with Apache Spark 2.0 Spark Summi...
Productionizing your Streaming Jobs
Learning spark ch05 - Loading and Saving Your Data
Ad

Viewers also liked (19)

PDF
Cm i sm_5
PPT
presentaion of hrd (luminous)
PDF
Cm i sm_7
PPT
Hondakinen gizartea
PPTX
Tics en el aula
PPTX
Eks about us
PDF
Ktm 1190 Adventure paper "La moda"
PPT
Hondakinen arazoak
PDF
PPTX
PDF
Cm i sm_sessio_3
PPTX
Egiving getting started july
PPTX
PPTX
Biografia beatriz
PDF
LSGlogoFinalAugust
PPTX
Roof covering.
PPTX
Juan sebastián elcano
PPTX
Paradigm shift in technology integration_CAGUIOA_III-A BSITE
PDF
merged_document_6
Cm i sm_5
presentaion of hrd (luminous)
Cm i sm_7
Hondakinen gizartea
Tics en el aula
Eks about us
Ktm 1190 Adventure paper "La moda"
Hondakinen arazoak
Cm i sm_sessio_3
Egiving getting started july
Biografia beatriz
LSGlogoFinalAugust
Roof covering.
Juan sebastián elcano
Paradigm shift in technology integration_CAGUIOA_III-A BSITE
merged_document_6
Ad

Similar to Using Spark for Timeseries Graph Analytics ved (20)

PPTX
GRAPH 101- GETTING STARTED WITH TITAN AND CASSANDRA
PDF
Titan and Cassandra at WellAware
PDF
Modeling the IoT with TitanDB and Cassandra
PDF
JanusGraph, Jupyter Meetup NYC
PDF
Graph Processing with Titan and Scylla
PDF
Scylla Summit 2016: Graph Processing with Titan and Scylla
PDF
Introduction to TitanDB
PPTX
Graph databases: Tinkerpop and Titan DB
PDF
Addressing performance issues in titan+cassandra
PDF
A Journey from Relational to Graph
PPTX
Titan NYC Meetup March 2014
PDF
TinkerPop: a story of graphs, DBs, and graph DBs
PPTX
Exploring Titan and Spark GraphX for Analyzing Time-Varying Electrical Networks
PPTX
BI, Reporting and Analytics on Apache Cassandra
PPTX
Gremlin Queries with DataStax Enterprise Graph
PDF
Titan - Graph Computing with Cassandra
PPTX
Hadoop summit 2017 enterprise graph analytics
PPTX
Enterprise large scale graph analytics and computing base on distribute graph...
PPTX
Hadoop Summit 2017 Enterprise Graph Analytics
GRAPH 101- GETTING STARTED WITH TITAN AND CASSANDRA
Titan and Cassandra at WellAware
Modeling the IoT with TitanDB and Cassandra
JanusGraph, Jupyter Meetup NYC
Graph Processing with Titan and Scylla
Scylla Summit 2016: Graph Processing with Titan and Scylla
Introduction to TitanDB
Graph databases: Tinkerpop and Titan DB
Addressing performance issues in titan+cassandra
A Journey from Relational to Graph
Titan NYC Meetup March 2014
TinkerPop: a story of graphs, DBs, and graph DBs
Exploring Titan and Spark GraphX for Analyzing Time-Varying Electrical Networks
BI, Reporting and Analytics on Apache Cassandra
Gremlin Queries with DataStax Enterprise Graph
Titan - Graph Computing with Cassandra
Hadoop summit 2017 enterprise graph analytics
Enterprise large scale graph analytics and computing base on distribute graph...
Hadoop Summit 2017 Enterprise Graph Analytics

Using Spark for Timeseries Graph Analytics ved

  • 1. Using Spark for Timeseries Graph Analytics SIGMOID 12 TH MEETUP -Ved Mulkalwar ved.mulkalwar@gmail.com 9564211606
  • 2. Content  Introduction to sample problem statement  Which Graph database is used and why  Installing Titan  Titan with Cassandra  The Gremlin Cassandra script: A way to store data in cassandra from Titan Gremlin  Accessing Titan with Spark
  • 3. Introducing which kind of time-series problems that can be solved using graph analytics and introducing the problem statement which I have been working on solving.
  • 4. The dynamics of a complex system is usually recorded in the form of time series. In recent years, the visibility graph algorithm and the horizontal visibility graph algorithm have been recently introduced as the mapping between time series and complex networks. Transforming time series into the graphs, the algorithms allows applying the methods of graph theoretical tools for characterizing time series, opening the possibility of building fruitful connections between time series analysis, nonlinear dynamics, and graph theory.
  • 5.  The problem statement which I have been working on:  Our initial goal was finding anomalies in the given timeseries. We started with slicing the data into small meaningfull parts so that the data becomes smaller and contiguous data points having the same property get clubbed together.  Later we created a graph with these parts and tried to find the parts having similar properties. Doing this will single out anomalies which was the goal.
  • 6. Which graph database I used and why i.e. Difference between graphX, neo4j and titan , why we used titan
  • 7. Titan vs graphX  The fundamental difference between Titan and GraphX lies in how they persistence data and how they process data, Titan by default persists data (vertices and edges and properties) to a distributed data store in the form of tables bound to a specific schema. This schema can be stored in Berkley DB tables, Cassandra tables or Hbase tables.  In the case of Titan the graph is stored as vertices in a vertex table and edges in an edge table.  GraphX has no real persistence layer (yet?), yes it can persist to HDFS files, but it cannot persist to a distributed datastore in a common schema like form as in Titan’s case.  A graph only exists in GraphX when it is loaded into memory off raw data and interpreted as Graph RDDs, Titan stores the graph permanently.  The other major difference between the two solutions is how they process graph data.  By default, GraphX solves queries via distributed processing on many nodes in parallel where possible as opposed to Titan processing pipelines on a single node. Titan can also take advantage of parallel processing via Faunus/HDFS if necessary.
  • 8. Neo4j vs titan  The primary difference between Titan and Neo4j is scalability: Titan can distribute the graph across multiple machines (using either Cassandra or Hbase as the storage backend) which does three things:  1) It allows Titan to store really, really large graphs by scaling out  2) It allows Titan to handle massive write loads  3) No single point of failure for very high availability. While Neo4j's HA setup gives you redundancy through replication, death of the master in such a setup leads to temporary service interruption while a new master is elected.  This allows Titan to scale to large deployments with thousands of concurrent users as demonstrated in the benchmark:  http://thinkaurelius.com/2012/08/06/titan-provides-real-time-big-graph-data/  Neo4j has been around much longer and is therefore more "mature" in some regards:  1) Integrated with an external lucene index gives it more sophisticated search capabilities  2) More integration points into other programming languages and development frameworks (e.g. spring)
  • 9. Intro to Titan graph db and how to use it with cassandra & spark
  • 10. Installing Titan  Downloaded the latest prebuilt version (0.9.0-M2) of Titan at s3.thinkaurelius.com/downloads/titan/titan-0.9.0-M2-hadoop1.zip.  Carry out the following steps to ensure that Titan is installed on each node in the cluster:
  • 11.  Now, use the Linux su (switch user) command to change to the root account, and move the install to the /usr/local/ location. Change the file and group membership of the install to the hadoop user, and create a symbolic link called titan so that the current Titan release can be referred to as the simplified path called /usr/local/titan:
  • 12. Titan with Cassandra  In this section, the Cassandra NoSQL database will be used as a storage mechanism for Titan. Although it does not use Hadoop, it is a large-scale, cluster-based database in its own right, and can scale to very large cluster sizes. A graph will be created, and stored in Cassandra using the Titan Gremlin shell. It will then be checked using Gremlin, and the stored data will be checked in Cassandra. The raw Titan Cassandra graph-based data will then be accessed from Spark. The first step then will be to install Cassandra on each node in the cluster.
  • 13.  Install Cassandra on all the nodes  Set up the Cassandra configuration under /etc/cassandra/conf by altering the cassandra.yaml file:  Install Cassandra on all the nodes  Set up the Cassandra configuration under /etc/cassandra/conf by altering the cassandra.yaml file:
  • 14.  Log files can be found under /var/log/cassandra, and the data is stored under /var/lib/cassandra. The nodetool command can be used on any Cassandra node to check the status of the Cassandra cluster:  The Cassandra CQL shell command called cqlsh can be used to access the cluster, and create objects. The shell is invoked next, and it shows that Cassandra version 2.0.13 is installed:
  • 15.  The Cassandra query language next shows a key space called keyspace1 that is being created and used via the CQL shell:
  • 16. The Gremlin Cassandra script  The interactive Titan Gremlin shell can be found within the bin directory of the Titan install, as shown here. Once started, it offers a Gremlin prompt:
  • 17.  The following script will be entered using the Gremlin shell. The first section of the script defines the configuration in terms of the storage (Cassandra), the port number, and the keyspace name that is to be used:
  • 18.  Next define the generic vertex properties' name and age for the graph to be created using the Management System. It then commits the management system changes:
  • 19.  Now, six vertices are added to the graph. Each one is given a numeric label to represent its identity. Each vertex is given an age and name value:
  • 20.  Finally, the graph edges are added to join the vertices together. Each edge has a relationship value. Once created, the changes are committed to store them to Titan, and therefore Cassandra:
  • 21.  This results in a simple person-based graph, shown in the following figure
  • 22.  This graph can then be tested in Titan via the Gremlin shell using a similar script to the previous one. Just enter the following script at the gremlin> prompt, as was shown previously. It uses the same initial six lines to create the titanGraph configuration, but it then creates a graph traversal variable g.  The graph traversal variable can be used to check the graph contents. Using the ValueMap option, it is possible to search for the graph nodes called Mike and Flo. They have been successfully found here:
  • 23.  Using the Cassandra CQL shell, and the Titan keyspace, it can be seen that a number of Titan tables have been created in Cassandra:  It can also be seen that the data exists in the edgestore table within Cassandra:  This assures us that a Titan graph has been created in the Gremlin shell, and is stored in Cassandra. Now, I will try to access the data from Spark.
  • 24. Accessing Titan with Spark  So far, Titan 0.9.0-M2 has been installed, and the graphs have successfully been created using Cassandra as backend storage options. These graphs have been created using Gremlin-based scripts. In this section, a properties file will be used via a Gremlin script to process a Titan-based graph using Apache Spark. Cassandra will be used with Titan as backend storage option.
  • 25. The following figure, shows the architecture used in this section.
  • 26.  Let us examine a properties file that can be used to connect to Cassandra as a storage backend for Titan. It contains sections for Cassandra, Apache Spark, and the Hadoop Gremlin configuration. My Cassandra properties file is called cassandra.properties, and it looks like this
  • 27. Into Titan code  The following necessary TinkerPop and Aurelius classes that will be used: