Lightning Fast Analytics with Cassandra and Spark
Download as PPTX, PDF0 likes568 views
This document provides an overview of using Apache Spark for analytics with DataStax Cassandra, highlighting Spark's speed, ease of use, and fault tolerance. It details the integration process, including setup, data access, and operations using the DataStax Cassandra Spark driver. The document also outlines the benefits of Spark on Cassandra for complex data analytics and stream processing capabilities.
![Accessing Data
CREATE TABLE test.words (word text PRIMARY KEY, count int);
INSERT INTO test.words (word, count) VALUES ('bar', 30);
INSERT INTO test.words (word, count) VALUES ('foo', 20);
// Use table as RDD
val rdd = sc.cassandraTable("test", "words")
// rdd: CassandraRDD[CassandraRow] = CassandraRDD[0]
rdd.toArray.foreach(println)
// CassandraRow[word: bar, count: 30]
// CassandraRow[word: foo, count: 20]
rdd.columnNames // Stream(word, count)
rdd.size // 2
val firstRow = rdd.first // firstRow: CassandraRow = CassandraRow[word: bar, count: 30]
firstRow.getInt("count") // Int = 30
* Accessing table above as RDD:](https://image.slidesharecdn.com/lightningfastanalyticswithspark-140712024848-phpapp02/85/Lightning-Fast-Analytics-with-Cassandra-and-Spark-14-320.jpg)
![Saving Data
val newRdd = sc.parallelize(Seq(("cat", 40), ("fox", 50)))
// newRdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[2]
newRdd.saveToCassandra("test", "words", Seq("word", "count"))
SELECT * FROM test.words;
word | count
------+-------
bar | 30
foo | 20
cat | 40
fox | 50
(4 rows)
* RDD above saved to Cassandra:](https://image.slidesharecdn.com/lightningfastanalyticswithspark-140712024848-phpapp02/85/Lightning-Fast-Analytics-with-Cassandra-and-Spark-15-320.jpg)
![Mapping Rows to Objects
CREATE TABLE test.cars (
id text PRIMARY KEY,
model text,
fuel_type text,
year int
);
case class Vehicle(
id: String,
model: String,
fuelType: String,
year: Int
)
sc.cassandraTable[Vehicle]("test", "cars").toArray
//Array(Vehicle(KF334L, Ford Mondeo, Petrol, 2009),
// Vehicle(MT8787, Hyundai x35, Diesel, 2011)
* Mapping rows to Scala Case Classes
* CQL underscore case column mapped to Scala camel case property
* Custom mapping functions (see docs)](https://image.slidesharecdn.com/lightningfastanalyticswithspark-140712024848-phpapp02/85/Lightning-Fast-Analytics-with-Cassandra-and-Spark-17-320.jpg)
Lightning Fast Analytics with Cassandra and Spark
- 1. ©2014 DataStax Confidential. Do not distribute without consent. timothy.vincent@datastax.com Tim Vincent Solution Engineer Lightning-fast analytics with Spark for Cassandra and DataStax Enterprise 1
- 2. What is Spark? * Apache Project since 2010 * Fast * 10x-100x faster than Hadoop MapReduce * In-memory storage * Single JVM process per node * Easy * Rich Scala, Java and Python APIs * 2x-5x less code * Interactive shell Analytic Analytic Search
- 5. API * Resilient Distributed Datasets (RDDs) * Collections of objects spread across a cluster * Stored in RAM or on Disk * Built through parallel transformations * Automatically rebuilt on failure * Operations * Transformations (e.g. map, filter, groupBy) * Actions (e.g. count, collect, save)
- 6. Operator Graph: Optimization and Fault Tolerance join filter groupBy Stage 3 Stage 1 Stage 2 A: B: C: D: E: F: map = Cached partition= RDD
- 7. Fast 0 500 1000 1500 2000 2500 3000 3500 4000 1 5 10 20 30 RunningTime(s) Number of Iterations Hadoop Spark 110 sec / iteration first iteration 80 sec further iterations 1 sec * Logistic Regression Performance
- 8. Why Spark on Cassandra? * Data model independent queries * Cross-table operations (JOIN, UNION, etc.) * Complex analytics (e.g. machine learning) * Data transformation, aggregation, etc. * Stream processing (coming soon) * Near real time
- 9. How to Spark on Cassandra? * DataStax Cassandra Spark driver * Open source: https://github.com/datastax/cassandra-driver-spark * Compatible with * Spark 0.9+ * Cassandra 2.0+ * DataStax Enterprise 4.5+
- 10. Analytics Workload Isolation Cassandra + Spark DC Cassandra Only DC Online App Analytical App Mixed Load Cassandra Cluster
- 11. Analytics High Availability * All nodes are Spark Workers * By default resilient to Worker failures * First Spark node promoted as Spark Master * Standby Master promoted on failure * Master HA available in DataStax Enterprise Spark Master Spark Standby Master Spark Worker
- 12. Cassandra Spark Driver * Cassandra tables exposed as Spark RDDs * Read from and write to Cassandra * Mapping of C* tables and rows to Scala objects * All Cassandra types supported and converted to Scala types * Server side data selection * Virtual Nodes support * Scala only driver for now
- 13. Connecting to Cassandra // Import Cassandra-specific functions on SparkContext and RDD objects import com.datastax.driver.spark._ // Spark connection options val conf = new SparkConf(true) .setMaster("spark://192.168.123.10:7077") .setAppName("cassandra-demo") .set("cassandra.connection.host", "192.168.123.10") // initial contact .set("cassandra.username", "cassandra") .set("cassandra.password", "cassandra") val sc = new SparkContext(conf)
- 14. Accessing Data CREATE TABLE test.words (word text PRIMARY KEY, count int); INSERT INTO test.words (word, count) VALUES ('bar', 30); INSERT INTO test.words (word, count) VALUES ('foo', 20); // Use table as RDD val rdd = sc.cassandraTable("test", "words") // rdd: CassandraRDD[CassandraRow] = CassandraRDD[0] rdd.toArray.foreach(println) // CassandraRow[word: bar, count: 30] // CassandraRow[word: foo, count: 20] rdd.columnNames // Stream(word, count) rdd.size // 2 val firstRow = rdd.first // firstRow: CassandraRow = CassandraRow[word: bar, count: 30] firstRow.getInt("count") // Int = 30 * Accessing table above as RDD:
- 15. Saving Data val newRdd = sc.parallelize(Seq(("cat", 40), ("fox", 50))) // newRdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[2] newRdd.saveToCassandra("test", "words", Seq("word", "count")) SELECT * FROM test.words; word | count ------+------- bar | 30 foo | 20 cat | 40 fox | 50 (4 rows) * RDD above saved to Cassandra:
- 16. Type Mapping CQL Type Scala Type ascii String bigint Long boolean Boolean counter Long decimal BigDecimal, java.math.BigDecimal double Double float Float inet java.net.InetAddress int Int list Vector, List, Iterable, Seq, IndexedSeq, java.util.List map Map, TreeMap, java.util.HashMap set Set, TreeSet, java.util.HashSet text, varchar String timestamp Long, java.util.Date, java.sql.Date, org.joda.time.DateTime timeuuid java.util.UUID uuid java.util.UUID varint BigInt, java.math.BigInteger *nullable values Option
- 17. Mapping Rows to Objects CREATE TABLE test.cars ( id text PRIMARY KEY, model text, fuel_type text, year int ); case class Vehicle( id: String, model: String, fuelType: String, year: Int ) sc.cassandraTable[Vehicle]("test", "cars").toArray //Array(Vehicle(KF334L, Ford Mondeo, Petrol, 2009), // Vehicle(MT8787, Hyundai x35, Diesel, 2011) * Mapping rows to Scala Case Classes * CQL underscore case column mapped to Scala camel case property * Custom mapping functions (see docs)
- 18. Server Side Data Selection * Reduce the amount of data transferred * Selecting columns * Selecting rows (by clustering columns and/or secondary indexes) sc.cassandraTable("test", "users").select("username").toArray.foreach(println) // CassandraRow{username: john} // CassandraRow{username: tom} sc.cassandraTable("test", "cars").select("model").where("color = ?", "black").toArray.foreach(println) // CassandraRow{model: Ford Mondeo} // CassandraRow{model: Hyundai x35}
- 19. Shark * SQL query engine on top of Spark * Not part of Apache Spark * Hive compatible (JDBC, UDFs, types, metadata, etc.) * Supports in-memory tables * Available as a part of DataStax Enterprise
- 20. Shark In-memory Tables CREATE TABLE CachedStocks TBLPROPERTIES ("shark.cache" = "true") AS SELECT * from PortfolioDemo.Stocks WHERE value > 95.0; OK Time taken: 1.215 seconds SELECT * FROM CachedStocks; OK MQT price 97.9270442241818 SII price 99.69238346610474 . . (123 additional prices) . PBG price 96.09162963505352 Time taken: 0.569 seconds
- 21. Spark SQL vs Shark Shark or Spark SQL Streaming ML Spark (General execution engine) Graph Cassandra Compatible
- 22. Questions?
Editor's Notes
- #7: Key thing to explain on this slide is that computation for 2nd iteration will not go beyond cached RDDs. So for example when F: requests second iteration it will not hit A: as long as data is in B:. We basically perform operation on A: and keep in B: RDD. Example of how Spark computes job stages. Boxes with solid outlines are RDDs. Partitions are shaded rectangles, in brownif they are already in memory. To run an action on RDD F, we build build stages at wide dependencies and pipeline narrow transformations inside each stage. In this case, stage 1’s output RDD is already in RAM, so we run stage 2 and then 3.