Spark + Cassandra
Download as PPTX, PDF0 likes763 views
This document discusses using Spark with Cassandra. It provides an overview of Spark and its Resilient Distributed Dataset (RDD) framework. It describes using the spark-cassandra-connector to expose Cassandra tables as RDDs to read from and write to Cassandra from Spark. It also gives an example of using Spark and Cassandra for bioinformatics by integrating with the ADAM genomics project.
![Saving To Cassandra
val variants: RDD[VariantContext] = sc.adamVCFLoad(args(0))
variants.flatMap(getVariant)
.saveToCassandra("adam", "variants", AllColumns)](https://image.slidesharecdn.com/sparkcassandra-141210200741-conversion-gate01/85/Spark-Cassandra-13-320.jpg)
![Querying Cassandra
val rdd = sc.cassandraTable("adam", "variants")
.map(r => (r.get[String]("allele"), 1L))
.reduceByKey(_ + _)
.map(r => (r._2, r._1))
.sortByKey(ascending = false)
rdd.collect()
.foreach(bc => println("%40st%d".format(bc._2, bc._1)))](https://image.slidesharecdn.com/sparkcassandra-141210200741-conversion-gate01/85/Spark-Cassandra-14-320.jpg)
Spark + Cassandra
- 1. Spark + Cassandra Carl Yeksigian DataStax
- 2. Spark -Fast large-scale data processing framework -Focused on in-memory workloads -Supports Java, Scala, and Python -Integrated machine learning support (MLlib) -Streaming support -Simple developer API
- 3. Resilient Distributed Dataset (RDD) -Presents a simple Collection API to the developer -Breaks full collection into partitions, which can be operated on independently -Knows how to recalculate itself if data is lost -Abstracts how to complete a job from the tasks
- 4. RDD
- 5. RDD API
- 6. Partitions -Partitions can be created so they are on the same machine as the data
- 7. Uses for Spark with Cassandra -Ad-hoc queries -Joins, Unions across tables -Rewriting tables -Machine Learning
- 8. spark-cassandra-connector DataStax OSS Project https://github.com/datastax/spark-cassandra-connector
- 9. Spark Cassandra Connector -Exposes Cassandra tables as RDDs -Read from and write to Cassandra -Data type mapping -Scala and Java support
- 10. Spark + Bioinformatics -ADAM is a bioinformatics project out of UC Berkeley AMPLab -Combines Spark + Parquet + Avro https://github.com/bigdatagenomics/adam http://bdgenomics.org/
- 11. Simple Variant case class Variant ( sampleid: String, referencename: String, location: Long, allele: String) create table adam.variants ( sampleid ascii, referencename ascii, location bigint, allele ascii)
- 12. Connecting to Cassandra import com.datastax.spark.connector._ // Spark connection options val conf = new SparkConf(true) .setMaster("spark://192.168.345.10:7077") .setAppName("cassandra-demo") .set("cassandra.connection.host", "192.168.345.10") val sc = new SparkContext(conf)
- 13. Saving To Cassandra val variants: RDD[VariantContext] = sc.adamVCFLoad(args(0)) variants.flatMap(getVariant) .saveToCassandra("adam", "variants", AllColumns)
- 14. Querying Cassandra val rdd = sc.cassandraTable("adam", "variants") .map(r => (r.get[String]("allele"), 1L)) .reduceByKey(_ + _) .map(r => (r._2, r._1)) .sortByKey(ascending = false) rdd.collect() .foreach(bc => println("%40st%d".format(bc._2, bc._1)))
- 15. Thanks Acknowledgements: Timothy Danford (AMPLab) Matt Massie (AMPLab) Frank Nothaft (AMPLab) Jeff Hammerbacher (Cloudera/Mt Sinai)