SlideShare a Scribd company logo

Cassandra and Spark SQL

Russell Spitzer, profile picture
Russell Spitzer

The document discusses using Spark SQL and the Spark SQL Thrift Server to access Cassandra data using SQL. It describes how the Thrift Server allows sharing of a single Spark context and cached data across clients. It also explains how predicates can be pushed down to Cassandra to optimize queries, though this does not always work for cases involving date comparisons.

Software
1 of 74
Downloaded 25 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
You don't need Functional Programming for Fun! Cassandra and SparkSQL
Russell (left) and Cara (right) • Software Engineer
 
 • Spark-Cassandra Integration since Spark 0.9 • Cassandra since Cassandra1.2 • 2 Year Scala Convert • Still not comfortable talking about Monads in public @Evanfchan
A Story in 3 Parts • Why SparkSQL? • The Spark SQL Thrift Server • Writing SQL for Spark
You have lots of options why Spark SQL? • Scala? • Java?
Spark is A Powerful Analytics Tool Built on Scala Distributed Analytics Platform with In Memory Capabilities Lots of new concepts:
 RDDs DataSets Streaming Serialization Functional Programming
Functional Programming Is Awesome Side-effect Free Functions Monads Easy Parallelization AnonymousFunctions Scala Async Models TypeMatching rdd.map(y => y+1) Endofunctors
Functional Programming can be Hard blah-blah blah Blah Easy blahilization baaaaah blahala Asybc blah Blahblahhing rdd.map(y => y+1) Aren't Endofunctors from ghostbusters? Endofunctors
Practical considerations when devoting time to a new Project. Compile Time Type Safety! Catalyst! Tungsten! We get to learn all sorts of fun new things! SBT is probably great! Usually Me Less Excitable Dev We ship next week
Spark SQL Provides A Familiar and Easy API Use SQL to access the Power of Spark
Spark Sql Provides A Familiar and Easy API Catalyst Codegen! Optimization! Predicate Pushdowns Distributed Work SQL
It still takes Scala/Java/Python/… Code. import org.apache.spark.sql.cassandra._
 val df = spark .read .cassandraFormat("tab", "ks") .load
 df.createTempView("tab")
 spark.sql("SELECT * FROM tab").show +---+---+---+ | k| c| v| +---+---+---+ | 1| 1| 1| | 1| 2| 2| Let me color code that by parts I like vs parts I don't like.
It still takes Scala/Java/Python/… Code. import org.apache.spark.sql.cassandra._
 val df = spark .read .cassandraFormat("tab", "ks") .load
 df.createTempView("tab")
 spark.sql("SELECT * FROM tab").show +---+---+---+ | k| c| v| +---+---+---+ | 1| 1| 1| | 1| 2| 2| Also, your import has an underscore in it..
For exploration we have the Spark-SQL Shell spark-sql> SELECT * FROM ks.tab; 1 2 2 1 3 3
For exploration we have the Spark-SQL Shell spark-sql> SELECT * FROM ks.tab; 1 2 2 1 3 3 SparkSession
For exploration we have the Spark-SQL Shell spark-sql> SELECT * FROM ks.tab; 1 2 2 1 3 3 SparkSession Executor Executor Executor Executor Executor
Not really good for multiple-users spark-sql> SELECT * FROM ks.tab; 1 2 2 1 3 3 SparkSession Executor Executor Executor Executor Executor
Enter Spark Thrift Server Spark Sql Thrift Server Executor Executor Executor Executor Executor JDBC Client JDBC ClientJDBC Client
The Spark Sql Thrift Server is a Spark Application • Built on HiveServer2 • Single Spark Context • Clients Communicate with it via JDBC • Can use all SparkSQL • Fair Scheduling • Clients can share Cached Resources • Security
The Spark Sql ThriftServer is a Spark Application • Built on HiveServer2 • Single Spark Context • Clients Communicate with it via JDBC • Can use all SparkSQL • Fair Scheduling • Clients can share Cached Resources • Security
Fair Scheduling is Sharing FIFO Time
Fair Scheduling is Sharing FIFO Time
Fair Scheduling is Sharing FIFO Time
Fair Scheduling is Sharing FIFO Time
Fair Scheduling is Sharing FIFO FAIR Time
Fair Scheduling is Sharing FIFO FAIR Time
Fair Scheduling is Sharing FIFO FAIR Time
SingleContext can Share Cached Data Spark Sql Thrift Server Executor Executor Executor Executor Executor cache TABLE today select * from ks.tab where date = today;
SingleContext can Share Cached Data Spark Sql Thrift Server Executor Executor Executor Executor Executor CACHED CACHED CACHED CACHED CACHED cache TABLE today select * from ks.tab where date = today;
SingleContext can Share Cached Data Spark Sql Thrift Server Executor Executor Executor Executor Executor CACHED CACHED CACHED CACHED CACHED cache TABLE today select * from ks.tab where date = today; SELECT * from TODAY where age > 5
How to use it Starts from the command line and can use all Spark Submit Args • ./sbin/start-thriftserver.sh • dse spark-sql-thriftserver start starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
How to use it Starts from the command line and can use all Spark Submit Args • ./sbin/start-thriftserver.sh • dse spark-sql-thriftserver start Use with all of your favorite Spark Packages like the Spark Cassandra Connector! --packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.2 --conf spark.cassandra.connection.host=127.0.0.1
Hive? Wait I though we were Doing Spark starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 Why does it say Hive everywhere? • Built on HiveServer2
A Brief History of the Spark Thrift Server • Thrift? • Hive?
They are not the Same Cassandra Thrift Hive Thrift
Have you heard of the
 "Ship of Theseus?" Time for a quick history More Greek stuff ..
When you replace all the parts of a thing Does it Remain the Same? Greek Boat
When you replace all the parts of a thing Does it Remain the Same? SharkServer Hive Parser Hive Optimization Map-Reduce Spark Execution JDBC Results
When you replace all the parts of a thing Does it Remain the Same? SharkServer
 ThriftServer Hive Parser Map-Reduce Spark Execution JDBC Results Hive Optimization
When you replace all the parts of a thing Does it Remain the Same? ThriftServer Hive Parser Catalyst Schema RDDs Spark Execution JDBC Results
When you replace all the parts of a thing Does it Remain the Same? ThriftServer Hive Parser Catalyst Dataframes Spark Execution JDBC Results
When you replace all the parts of a thing Does it Remain the Same? ThriftServer Hive Parser Catalyst DataSets Spark Execution JDBC Results
When you replace all the parts of a thing Does it Remain the Same? ThriftServer Spark Parser Catalyst DataSets Spark Execution JDBC Results
Almost all Spark now ThriftServer Spark Parser Catalyst DataSets Spark Execution JDBC Results
Connecting with Beeline (JDBC Client) ./bin/beeline dse beeline !connect jdbc:hive2://localhost:10000 Even More Hive!
Connect Tableau to Cassandra
The Full JDBC/ODBC Ecosystem Can Connect to ThriftServer
Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server ALL THE DATA
Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server ALL THE DATAOOM
Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server
 spark.sql.thriftServer.incrementalCollect=true ALL THE DATASpark Partition 1 Spark Partition 2 Spark Partition 3
Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server
 spark.sql.thriftServer.incrementalCollect=true ALL THE DATA Spark Partition 1 Spark Partition 2 Spark Partition 3
Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server
 spark.sql.thriftServer.incrementalCollect=true ALL THE DATA Spark Partition 1 Spark Partition 2 Spark Partition 3
Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server
 spark.sql.thriftServer.incrementalCollect=true ALL THE DATA Spark Partition 1Spark Partition 2 Spark Partition 3
Getting things done with SQL • Registering Sources • Writing to Tables • Examining Query Plans • Debugging Predicate pushdowns • Caching Views
Registering Sources using SQL CREATE TEMPORARY VIEW words USING format.goes.here OPTIONS ( key "value"
 )
Registering Sources using SQL CREATE TEMPORARY VIEW words USING org.apache.spark.sql.cassandra OPTIONS ( table "tab", keyspace "ks") Not a single monad…
CREATE TEMPORARY VIEW words USING org.apache.spark.sql.cassandra OPTIONS ( table "tab", keyspace "ks") Registering Sources using SQL CassandraSourceRelation
We Can Still Use a HiveMetaStore DSE auto registers C* Tables in a C* based Metastore MetaStore Thrift Server
Writing DataFrames using SQL INSERT INTO arrow SELECT * FROM words; CassandraSourceRelation words read CassandraSourceRelation arrow
 write
Explain to Analyze Query Plans EXPLAIN SELECT * FROM arrow WHERE C > 2;
 
 Scan org.apache.spark.sql.cassandra.CassandraSourceRelation@6069193a [k#18,c#19,v#20] 
 PushedFilters: [IsNotNull(c), GreaterThan(c,2)], 
 ReadSchema: struct<k:int,c:int,v:int> We can analyze the inside of the Catalyst just like with Scala/Java/…
Predicates get Pushed Down Automatically EXPLAIN SELECT * FROM arrow WHERE C > 2;
 
 Scan org.apache.spark.sql.cassandra.CassandraSourceRelation@6069193a [k#18,c#19,v#20] 
 PushedFilters: [IsNotNull(c), GreaterThan(c,2)], 
 ReadSchema: struct<k:int,c:int,v:int> CassandraSourceRelation Filter [GreaterThan(c,2)]
EXPLAIN SELECT * FROM arrow WHERE C > 2;
 
 Scan org.apache.spark.sql.cassandra.CassandraSourceRelation@6069193a [k#18,c#19,v#20] 
 PushedFilters: [IsNotNull(c), GreaterThan(c,2)], 
 ReadSchema: struct<k:int,c:int,v:int> CassandraSourceRelation Filter [GreaterThan(c,2)] Internal Request to Cassandra: CQL SELECT * FROM ks.bat WHERE C > 2 Automatic Pushdowns! Predicates get Pushed Down Automatically
Common Cases where Predicates Don't Push SELECT * from troubles WHERE c < '2017-05-27'
 *Filter (cast(c#76 as string) < 2017-05-27) +- *Scan CassandraSourceRelation@53e82b30 [k#75,c#76,v#77] PushedFilters: [IsNotNull(c)], ReadSchema: struct<k:int,c:date,v:int> Why is my date clustering column not being pushed down.
Common Cases where Predicates Don't Push CassandraSourceRelation Filter [LessThan(c,'2017-05-27')] SELECT * from troubles WHERE c < '2017-05-27'
 *Filter (cast(c#76 as string) < 2017-05-27) +- *Scan CassandraSourceRelation@53e82b30 [k#75,c#76,v#77] PushedFilters: [IsNotNull(c)], ReadSchema: struct<k:int,c:date,v:int>
Common Cases where Predicates Don't Push CassandraSourceRelation
 ReadSchema: struct<k:int,c:date,v:int> Filter [LessThan(c,'2017-05-27')] Date != String SELECT * from troubles WHERE c < '2017-05-27'
 *Filter (cast(c#76 as string) < 2017-05-27) +- *Scan CassandraSourceRelation@53e82b30 [k#75,c#76,v#77] PushedFilters: [IsNotNull(c)], ReadSchema: struct<k:int,c:date,v:int>
Make Sure we Cast Correctly EXPLAIN SELECT * from troubles WHERE c < cast('2017-05-27' as date);
 *Scan C*Relation PushedFilters: [IsNotNull(c), LessThan(c, 2017-05-27)] CassandraSourceRelation
 ReadSchema: struct<k:int,c:date,v:int> Filter [LessThan(c,Date('2017-05-27'))] Date == Date
Make Sure we Cast Correctly EXPLAIN SELECT * from troubles WHERE c < cast('2017-05-27' as date);
 *Scan C*Relation PushedFilters: [IsNotNull(c), LessThan(c, 2017-05-27)] CassandraSourceRelation
 ReadSchema: struct<k:int,c:date,v:int> Filter [LessThan(c,Date('2017-05-27'))] Automatic Pushdowns!
DSE Search Automatic Pushdowns! EXPLAIN SELECT * from troubles WHERE v < 6;
 *Scan C*Relation PushedFilters: [IsNotNull(v), LessThan(v, 6)] CassandraSourceRelation
 ReadSchema: struct<k:int,c:date,v:int> Solr_Query
DSE Search Automatic Pushdowns!
DSE Search Automatic Pushdowns! Count Happens in the Index DSE
 Continuous Paging
Cache a whole table CassandraSourceRelation InMemoryRelation CACHE TABLE ks.tab;
 explain SELECT * FROM ks.tab; == Physical Plan == InMemoryTableScan [k#0, c#1, v#2] : +- InMemoryRelation StorageLevel(disk, memory, deserialized, 1 replicas), `ks`.`tab` : : +- *Scan CassandraSourceRelation
Uncache CassandraSourceRelation UNCACHE TABLE ks.tab; explain SELECT * FROM ks.tab; == Physical Plan == *Scan CassandraSourceRelation
Cache a fraction of Data CassandraSourceRelation CACHE TABLE somedata SELECT * FROM ks.tab WHERE c > 2;
 explain SELECT * from somedata; == Physical Plan == InMemoryTableScan : +- InMemoryRelation `somedata` : : +- *Scan CassandraSourceRelation PushedFilters: [IsNotNull(c), GreaterThan(c,2)] Filter [GreaterThan(c,2)] InMemoryRelation somedata
Let this be a starting point • https://github.com/datastax/spark-cassandra-connector • https://github.com/datastax/spark-cassandra-connector/blob/master/doc/14_data_frames.md • https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-thrift-server.html • http://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/spark/sparkSqlThriftServer.html • https://spark.apache.org/docs/latest/sql-programming-guide.html#distributed-sql-engine • https://www.datastax.com/dev/blog/dse-5-1-automatic-optimization-of-spark-sql-queries-using-dse-search • https://www.datastax.com/dev/blog/dse-continuous-paging-tuning-and-support-guide
Thank You. http://www.russellspitzer.com/
 @RussSpitzer Come chat with us at DataStax Academy: 
 https://academy.datastax.com/slack

More Related Content

PDF
Spark And Cassandra: 2 Fast, 2 Furious
Jen Aman
 
PDF
Analytics with Cassandra & Spark
Matthias Niehoff
 
PDF
Analyzing Time Series Data with Apache Spark and Cassandra
Patrick McFadin
 
PDF
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
Helena Edelson
 
PDF
Maximum Overdrive: Tuning the Spark Cassandra Connector (Russell Spitzer, Dat...
DataStax
 
PDF
OLAP with Cassandra and Spark
Evan Chan
 
PDF
Homologous Apache Spark Clusters Using Nomad with Alex Dadgar
Databricks
 
PPTX
Real time data pipeline with spark streaming and cassandra with mesos
Rahul Kumar
 
Spark And Cassandra: 2 Fast, 2 Furious
Jen Aman
 
Analytics with Cassandra & Spark
Matthias Niehoff
 
Analyzing Time Series Data with Apache Spark and Cassandra
Patrick McFadin
 
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
Helena Edelson
 
Maximum Overdrive: Tuning the Spark Cassandra Connector (Russell Spitzer, Dat...
DataStax
 
OLAP with Cassandra and Spark
Evan Chan
 
Homologous Apache Spark Clusters Using Nomad with Alex Dadgar
Databricks
 
Real time data pipeline with spark streaming and cassandra with mesos
Rahul Kumar
 

What's hot (20)

PDF
Cassandra and SparkSQL: You Don't Need Functional Programming for Fun with Ru...
Databricks
 
PPTX
Real time Analytics with Apache Kafka and Apache Spark
Rahul Jain
 
PDF
Lambda architecture
Szilveszter Molnár
 
PDF
Spark Streaming with Cassandra
Jacek Lewandowski
 
PDF
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
Helena Edelson
 
PDF
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
DataStax Academy
 
PPTX
Real time data viz with Spark Streaming, Kafka and D3.js
Ben Laird
 
PDF
Spark streaming , Spark SQL
Yousun Jeong
 
PPTX
Processing Large Data with Apache Spark -- HasGeek
Venkata Naga Ravi
 
PPTX
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
spark-project
 
PDF
Apache cassandra & apache spark for time series data
Patrick McFadin
 
PDF
Cassandra spark connector
Duyhai Doan
 
POTX
Apache Spark Streaming: Architecture and Fault Tolerance
Sachin Aggarwal
 
PDF
Dive into Spark Streaming
Gerard Maas
 
PDF
Data Analytics with Apache Spark and Cassandra
Gerard Maas
 
PPTX
Lambda Architecture: The Best Way to Build Scalable and Reliable Applications!
Tugdual Grall
 
PDF
Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)
Spark Summit
 
PDF
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
Nathan Bijnens
 
PDF
Spark Tuning For Enterprise System Administrators, Spark Summit East 2016
Anya Bida
 
PDF
Scaling Data Analytics Workloads on Databricks
Databricks
 
Cassandra and SparkSQL: You Don't Need Functional Programming for Fun with Ru...
Databricks
 
Real time Analytics with Apache Kafka and Apache Spark
Rahul Jain
 
Lambda architecture
Szilveszter Molnár
 
Spark Streaming with Cassandra
Jacek Lewandowski
 
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
Helena Edelson
 
C* Summit 2013: Real-time Analytics using Cassandra, Spark and Shark by Evan ...
DataStax Academy
 
Real time data viz with Spark Streaming, Kafka and D3.js
Ben Laird
 
Spark streaming , Spark SQL
Yousun Jeong
 
Processing Large Data with Apache Spark -- HasGeek
Venkata Naga Ravi
 
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
spark-project
 
Apache cassandra & apache spark for time series data
Patrick McFadin
 
Cassandra spark connector
Duyhai Doan
 
Apache Spark Streaming: Architecture and Fault Tolerance
Sachin Aggarwal
 
Dive into Spark Streaming
Gerard Maas
 
Data Analytics with Apache Spark and Cassandra
Gerard Maas
 
Lambda Architecture: The Best Way to Build Scalable and Reliable Applications!
Tugdual Grall
 
Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)
Spark Summit
 
A real-time (lambda) architecture using Hadoop & Storm (NoSQL Matters Cologne...
Nathan Bijnens
 
Spark Tuning For Enterprise System Administrators, Spark Summit East 2016
Anya Bida
 
Scaling Data Analytics Workloads on Databricks
Databricks
 
Ad

Similar to Cassandra and Spark SQL (20)

PPTX
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
Simplilearn
 
PDF
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Helena Edelson
 
PDF
Jumpstart on Apache Spark 2.2 on Databricks
Databricks
 
PDF
Jump Start on Apache® Spark™ 2.x with Databricks
Databricks
 
PDF
실시간 Streaming using Spark and Kafka 강의교재
hkyoon2
 
PDF
Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3
Databricks
 
PPTX
Spark from the Surface
Josi Aranda
 
PDF
Jump Start with Apache Spark 2.0 on Databricks
Databricks
 
PPTX
Intro to Spark
Kyle Burke
 
PPTX
Incorta spark integration
Dylan Wan
 
PDF
Harnessing Spark and Cassandra with Groovy
Steve Pember
 
PDF
Headaches and Breakthroughs in Building Continuous Applications
Databricks
 
PPTX
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Landon Robinson
 
PPTX
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
Chris Fregly
 
PPTX
Productionizing Spark and the REST Job Server- Evan Chan
Spark Summit
 
PPTX
Spark core
Prashant Gupta
 
PPT
Apache spark-melbourne-april-2015-meetup
Ned Shawa
 
PPTX
Intro to Apache Spark
Robert Sanders
 
PPTX
Intro to Apache Spark
clairvoyantllc
 
PDF
Productionizing Spark and the Spark Job Server
Evan Chan
 
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
Simplilearn
 
Streaming Big Data with Spark, Kafka, Cassandra, Akka & Scala (from webinar)
Helena Edelson
 
Jumpstart on Apache Spark 2.2 on Databricks
Databricks
 
Jump Start on Apache® Spark™ 2.x with Databricks
Databricks
 
실시간 Streaming using Spark and Kafka 강의교재
hkyoon2
 
Spark Saturday: Spark SQL & DataFrame Workshop with Apache Spark 2.3
Databricks
 
Spark from the Surface
Josi Aranda
 
Jump Start with Apache Spark 2.0 on Databricks
Databricks
 
Intro to Spark
Kyle Burke
 
Incorta spark integration
Dylan Wan
 
Harnessing Spark and Cassandra with Groovy
Steve Pember
 
Headaches and Breakthroughs in Building Continuous Applications
Databricks
 
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Landon Robinson
 
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
Chris Fregly
 
Productionizing Spark and the REST Job Server- Evan Chan
Spark Summit
 
Spark core
Prashant Gupta
 
Apache spark-melbourne-april-2015-meetup
Ned Shawa
 
Intro to Apache Spark
Robert Sanders
 
Intro to Apache Spark
clairvoyantllc
 
Productionizing Spark and the Spark Job Server
Evan Chan
 
Ad

More from Russell Spitzer (10)

PDF
Tale of Two Graph Frameworks: Graph Frames and Tinkerpop
Russell Spitzer
 
PPTX
Maximum Overdrive: Tuning the Spark Cassandra Connector
Russell Spitzer
 
PDF
Spark and Cassandra 2 Fast 2 Furious
Russell Spitzer
 
PDF
Spark Cassandra Connector: Past, Present, and Future
Russell Spitzer
 
PDF
Spark Cassandra Connector Dataframes
Russell Spitzer
 
PDF
Cassandra and Spark: Optimizing for Data Locality
Russell Spitzer
 
PDF
Cassandra and IoT
Russell Spitzer
 
PDF
Zero to Streaming: Spark and Cassandra
Russell Spitzer
 
PDF
Cassandra Fundamentals - C* 2.0
Russell Spitzer
 
PDF
Escape From Hadoop: Spark One Liners for C* Ops
Russell Spitzer
 
Tale of Two Graph Frameworks: Graph Frames and Tinkerpop
Russell Spitzer
 
Maximum Overdrive: Tuning the Spark Cassandra Connector
Russell Spitzer
 
Spark and Cassandra 2 Fast 2 Furious
Russell Spitzer
 
Spark Cassandra Connector: Past, Present, and Future
Russell Spitzer
 
Spark Cassandra Connector Dataframes
Russell Spitzer
 
Cassandra and Spark: Optimizing for Data Locality
Russell Spitzer
 
Cassandra and IoT
Russell Spitzer
 
Zero to Streaming: Spark and Cassandra
Russell Spitzer
 
Cassandra Fundamentals - C* 2.0
Russell Spitzer
 
Escape From Hadoop: Spark One Liners for C* Ops
Russell Spitzer
 

Recently uploaded (20)

PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
PDF
Nekopoi APK 2025 free lastest update
umarali35kp
 
PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PDF
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
PPTX
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
PPTX
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
PDF
System and Network Administraation Chapter 3
jaramharo
 
PDF
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
PDF
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
PDF
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PDF
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
PPTX
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
PPTX
history of c programming in notes for students .pptx
Vijayakumari829030
 
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
Nekopoi APK 2025 free lastest update
umarali35kp
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
Transform Your Business with a Software ERP System
Canada Software Company
 
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
System and Network Administraation Chapter 3
jaramharo
 
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
PTS Company Brochure 2025 (1).pdf.......
pts464036
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
history of c programming in notes for students .pptx
Vijayakumari829030
 

Cassandra and Spark SQL

  • 1. You don't need Functional Programming for Fun! Cassandra and SparkSQL
  • 2. Russell (left) and Cara (right) • Software Engineer
 
 • Spark-Cassandra Integration since Spark 0.9 • Cassandra since Cassandra1.2 • 2 Year Scala Convert • Still not comfortable talking about Monads in public @Evanfchan
  • 3. A Story in 3 Parts • Why SparkSQL? • The Spark SQL Thrift Server • Writing SQL for Spark
  • 4. You have lots of options why Spark SQL? • Scala? • Java?
  • 5. Spark is A Powerful Analytics Tool Built on Scala Distributed Analytics Platform with In Memory Capabilities Lots of new concepts:
 RDDs DataSets Streaming Serialization Functional Programming
  • 6. Functional Programming Is Awesome Side-effect Free Functions Monads Easy Parallelization AnonymousFunctions Scala Async Models TypeMatching rdd.map(y => y+1) Endofunctors
  • 7. Functional Programming can be Hard blah-blah blah Blah Easy blahilization baaaaah blahala Asybc blah Blahblahhing rdd.map(y => y+1) Aren't Endofunctors from ghostbusters? Endofunctors
  • 8. Practical considerations when devoting time to a new Project. Compile Time Type Safety! Catalyst! Tungsten! We get to learn all sorts of fun new things! SBT is probably great! Usually Me Less Excitable Dev We ship next week
  • 9. Spark SQL Provides A Familiar and Easy API Use SQL to access the Power of Spark
  • 10. Spark Sql Provides A Familiar and Easy API Catalyst Codegen! Optimization! Predicate Pushdowns Distributed Work SQL
  • 11. It still takes Scala/Java/Python/… Code. import org.apache.spark.sql.cassandra._
 val df = spark .read .cassandraFormat("tab", "ks") .load
 df.createTempView("tab")
 spark.sql("SELECT * FROM tab").show +---+---+---+ | k| c| v| +---+---+---+ | 1| 1| 1| | 1| 2| 2| Let me color code that by parts I like vs parts I don't like.
  • 12. It still takes Scala/Java/Python/… Code. import org.apache.spark.sql.cassandra._
 val df = spark .read .cassandraFormat("tab", "ks") .load
 df.createTempView("tab")
 spark.sql("SELECT * FROM tab").show +---+---+---+ | k| c| v| +---+---+---+ | 1| 1| 1| | 1| 2| 2| Also, your import has an underscore in it..
  • 13. For exploration we have the Spark-SQL Shell spark-sql> SELECT * FROM ks.tab; 1 2 2 1 3 3
  • 14. For exploration we have the Spark-SQL Shell spark-sql> SELECT * FROM ks.tab; 1 2 2 1 3 3 SparkSession
  • 15. For exploration we have the Spark-SQL Shell spark-sql> SELECT * FROM ks.tab; 1 2 2 1 3 3 SparkSession Executor Executor Executor Executor Executor
  • 16. Not really good for multiple-users spark-sql> SELECT * FROM ks.tab; 1 2 2 1 3 3 SparkSession Executor Executor Executor Executor Executor
  • 17. Enter Spark Thrift Server Spark Sql Thrift Server Executor Executor Executor Executor Executor JDBC Client JDBC ClientJDBC Client
  • 18. The Spark Sql Thrift Server is a Spark Application • Built on HiveServer2 • Single Spark Context • Clients Communicate with it via JDBC • Can use all SparkSQL • Fair Scheduling • Clients can share Cached Resources • Security
  • 19. The Spark Sql ThriftServer is a Spark Application • Built on HiveServer2 • Single Spark Context • Clients Communicate with it via JDBC • Can use all SparkSQL • Fair Scheduling • Clients can share Cached Resources • Security
  • 20. Fair Scheduling is Sharing FIFO Time
  • 21. Fair Scheduling is Sharing FIFO Time
  • 22. Fair Scheduling is Sharing FIFO Time
  • 23. Fair Scheduling is Sharing FIFO Time
  • 24. Fair Scheduling is Sharing FIFO FAIR Time
  • 25. Fair Scheduling is Sharing FIFO FAIR Time
  • 26. Fair Scheduling is Sharing FIFO FAIR Time
  • 27. SingleContext can Share Cached Data Spark Sql Thrift Server Executor Executor Executor Executor Executor cache TABLE today select * from ks.tab where date = today;
  • 28. SingleContext can Share Cached Data Spark Sql Thrift Server Executor Executor Executor Executor Executor CACHED CACHED CACHED CACHED CACHED cache TABLE today select * from ks.tab where date = today;
  • 29. SingleContext can Share Cached Data Spark Sql Thrift Server Executor Executor Executor Executor Executor CACHED CACHED CACHED CACHED CACHED cache TABLE today select * from ks.tab where date = today; SELECT * from TODAY where age > 5
  • 30. How to use it Starts from the command line and can use all Spark Submit Args • ./sbin/start-thriftserver.sh • dse spark-sql-thriftserver start starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
  • 31. How to use it Starts from the command line and can use all Spark Submit Args • ./sbin/start-thriftserver.sh • dse spark-sql-thriftserver start Use with all of your favorite Spark Packages like the Spark Cassandra Connector! --packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.2 --conf spark.cassandra.connection.host=127.0.0.1
  • 32. Hive? Wait I though we were Doing Spark starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 Why does it say Hive everywhere? • Built on HiveServer2
  • 33. A Brief History of the Spark Thrift Server • Thrift? • Hive?
  • 34. They are not the Same Cassandra Thrift Hive Thrift
  • 35. Have you heard of the
 "Ship of Theseus?" Time for a quick history More Greek stuff ..
  • 36. When you replace all the parts of a thing Does it Remain the Same? Greek Boat
  • 37. When you replace all the parts of a thing Does it Remain the Same? SharkServer Hive Parser Hive Optimization Map-Reduce Spark Execution JDBC Results
  • 38. When you replace all the parts of a thing Does it Remain the Same? SharkServer
 ThriftServer Hive Parser Map-Reduce Spark Execution JDBC Results Hive Optimization
  • 39. When you replace all the parts of a thing Does it Remain the Same? ThriftServer Hive Parser Catalyst Schema RDDs Spark Execution JDBC Results
  • 40. When you replace all the parts of a thing Does it Remain the Same? ThriftServer Hive Parser Catalyst Dataframes Spark Execution JDBC Results
  • 41. When you replace all the parts of a thing Does it Remain the Same? ThriftServer Hive Parser Catalyst DataSets Spark Execution JDBC Results
  • 42. When you replace all the parts of a thing Does it Remain the Same? ThriftServer Spark Parser Catalyst DataSets Spark Execution JDBC Results
  • 43. Almost all Spark now ThriftServer Spark Parser Catalyst DataSets Spark Execution JDBC Results
  • 44. Connecting with Beeline (JDBC Client) ./bin/beeline dse beeline !connect jdbc:hive2://localhost:10000 Even More Hive!
  • 45. Connect Tableau to Cassandra
  • 46. The Full JDBC/ODBC Ecosystem Can Connect to ThriftServer
  • 47. Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server ALL THE DATA
  • 48. Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server ALL THE DATAOOM
  • 49. Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server
 spark.sql.thriftServer.incrementalCollect=true ALL THE DATASpark Partition 1 Spark Partition 2 Spark Partition 3
  • 50. Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server
 spark.sql.thriftServer.incrementalCollect=true ALL THE DATA Spark Partition 1 Spark Partition 2 Spark Partition 3
  • 51. Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server
 spark.sql.thriftServer.incrementalCollect=true ALL THE DATA Spark Partition 1 Spark Partition 2 Spark Partition 3
  • 52. Incremental Collect - Because some BI Tools are Mean SELECT * FROM TABLE Spark Sql Thrift Server
 spark.sql.thriftServer.incrementalCollect=true ALL THE DATA Spark Partition 1Spark Partition 2 Spark Partition 3
  • 53. Getting things done with SQL • Registering Sources • Writing to Tables • Examining Query Plans • Debugging Predicate pushdowns • Caching Views
  • 54. Registering Sources using SQL CREATE TEMPORARY VIEW words USING format.goes.here OPTIONS ( key "value"
 )
  • 55. Registering Sources using SQL CREATE TEMPORARY VIEW words USING org.apache.spark.sql.cassandra OPTIONS ( table "tab", keyspace "ks") Not a single monad…
  • 57. We Can Still Use a HiveMetaStore DSE auto registers C* Tables in a C* based Metastore MetaStore Thrift Server
  • 58. Writing DataFrames using SQL INSERT INTO arrow SELECT * FROM words; CassandraSourceRelation words read CassandraSourceRelation arrow
 write
  • 59. Explain to Analyze Query Plans EXPLAIN SELECT * FROM arrow WHERE C > 2;
 
 Scan org.apache.spark.sql.cassandra.CassandraSourceRelation@6069193a [k#18,c#19,v#20] 
 PushedFilters: [IsNotNull(c), GreaterThan(c,2)], 
 ReadSchema: struct<k:int,c:int,v:int> We can analyze the inside of the Catalyst just like with Scala/Java/…
  • 60. Predicates get Pushed Down Automatically EXPLAIN SELECT * FROM arrow WHERE C > 2;
 
 Scan org.apache.spark.sql.cassandra.CassandraSourceRelation@6069193a [k#18,c#19,v#20] 
 PushedFilters: [IsNotNull(c), GreaterThan(c,2)], 
 ReadSchema: struct<k:int,c:int,v:int> CassandraSourceRelation Filter [GreaterThan(c,2)]
  • 62. Common Cases where Predicates Don't Push SELECT * from troubles WHERE c < '2017-05-27'
 *Filter (cast(c#76 as string) < 2017-05-27) +- *Scan CassandraSourceRelation@53e82b30 [k#75,c#76,v#77] PushedFilters: [IsNotNull(c)], ReadSchema: struct<k:int,c:date,v:int> Why is my date clustering column not being pushed down.
  • 63. Common Cases where Predicates Don't Push CassandraSourceRelation Filter [LessThan(c,'2017-05-27')] SELECT * from troubles WHERE c < '2017-05-27'
 *Filter (cast(c#76 as string) < 2017-05-27) +- *Scan CassandraSourceRelation@53e82b30 [k#75,c#76,v#77] PushedFilters: [IsNotNull(c)], ReadSchema: struct<k:int,c:date,v:int>
  • 64. Common Cases where Predicates Don't Push CassandraSourceRelation
 ReadSchema: struct<k:int,c:date,v:int> Filter [LessThan(c,'2017-05-27')] Date != String SELECT * from troubles WHERE c < '2017-05-27'
 *Filter (cast(c#76 as string) < 2017-05-27) +- *Scan CassandraSourceRelation@53e82b30 [k#75,c#76,v#77] PushedFilters: [IsNotNull(c)], ReadSchema: struct<k:int,c:date,v:int>
  • 65. Make Sure we Cast Correctly EXPLAIN SELECT * from troubles WHERE c < cast('2017-05-27' as date);
 *Scan C*Relation PushedFilters: [IsNotNull(c), LessThan(c, 2017-05-27)] CassandraSourceRelation
 ReadSchema: struct<k:int,c:date,v:int> Filter [LessThan(c,Date('2017-05-27'))] Date == Date
  • 66. Make Sure we Cast Correctly EXPLAIN SELECT * from troubles WHERE c < cast('2017-05-27' as date);
 *Scan C*Relation PushedFilters: [IsNotNull(c), LessThan(c, 2017-05-27)] CassandraSourceRelation
 ReadSchema: struct<k:int,c:date,v:int> Filter [LessThan(c,Date('2017-05-27'))] Automatic Pushdowns!
  • 67. DSE Search Automatic Pushdowns! EXPLAIN SELECT * from troubles WHERE v < 6;
 *Scan C*Relation PushedFilters: [IsNotNull(v), LessThan(v, 6)] CassandraSourceRelation
 ReadSchema: struct<k:int,c:date,v:int> Solr_Query
  • 68. DSE Search Automatic Pushdowns!
  • 69. DSE Search Automatic Pushdowns! Count Happens in the Index DSE
 Continuous Paging
  • 70. Cache a whole table CassandraSourceRelation InMemoryRelation CACHE TABLE ks.tab;
 explain SELECT * FROM ks.tab; == Physical Plan == InMemoryTableScan [k#0, c#1, v#2] : +- InMemoryRelation StorageLevel(disk, memory, deserialized, 1 replicas), `ks`.`tab` : : +- *Scan CassandraSourceRelation
  • 72. Cache a fraction of Data CassandraSourceRelation CACHE TABLE somedata SELECT * FROM ks.tab WHERE c > 2;
 explain SELECT * from somedata; == Physical Plan == InMemoryTableScan : +- InMemoryRelation `somedata` : : +- *Scan CassandraSourceRelation PushedFilters: [IsNotNull(c), GreaterThan(c,2)] Filter [GreaterThan(c,2)] InMemoryRelation somedata
  • 73. Let this be a starting point • https://github.com/datastax/spark-cassandra-connector • https://github.com/datastax/spark-cassandra-connector/blob/master/doc/14_data_frames.md • https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-thrift-server.html • http://docs.datastax.com/en/dse/5.1/dse-dev/datastax_enterprise/spark/sparkSqlThriftServer.html • https://spark.apache.org/docs/latest/sql-programming-guide.html#distributed-sql-engine • https://www.datastax.com/dev/blog/dse-5-1-automatic-optimization-of-spark-sql-queries-using-dse-search • https://www.datastax.com/dev/blog/dse-continuous-paging-tuning-and-support-guide
  • 74. Thank You. http://www.russellspitzer.com/
 @RussSpitzer Come chat with us at DataStax Academy: 
 https://academy.datastax.com/slack