Spark Summit EU talk by John Musser
1 like992 views
This document provides lessons learned from optimizing Apache Spark for NoSQL databases like Riak. Some key lessons include: 1. Parallelizing operations whenever possible to avoid overloading Riak with too many direct key-based gets or secondary index queries. 2. Being smart about data mapping between NoSQL data structures and Spark DataFrames/RDDs for efficient processing. 3. Optimizing performance at all levels from the network protocol to data locality optimizations. 4. Being flexible in supporting multiple languages and deployment environments for Spark and NoSQL integrations.
.queryAll()
Code: Key/Value Query](https://image.slidesharecdn.com/3sjohnmusser-161103183222/85/Spark-Summit-EU-talk-by-John-Musser-55-320.jpg)
.queryBucketKeys("Alice", "Bob", "Charlie")
val rdd =
sc.riakBucket[String](kv_bucket_name)
.query2iRange("myIndex", 1L, 5000L)
Code: Key/Value Query
Query by 2i Strings
val rdd =
sc.riakBucket[String](kv_bucket_name)
.query2iKeys("dailyDataIndx", ”Jan", ”Feb”)](https://image.slidesharecdn.com/3sjohnmusser-161103183222/85/Spark-Summit-EU-talk-by-John-Musser-56-320.jpg)
.queryAll()
val df = riakRdd.toDF()
Code](https://image.slidesharecdn.com/3sjohnmusser-161103183222/85/Spark-Summit-EU-talk-by-John-Musser-62-320.jpg)
![Python
import pyspark_riak
conf = pyspark.SparkConf().setAppName("My Spark Riak App")
conf.set("spark.riak.connection.host", "127.0.0.1:8087")
sc = pyspark.SparkContext(conf)
pyspark_riak.riak_context(sc)
my_data = [{'key0':{'data': 0}}, {'key1':{'data': 1}}]
kv_write_rdd = sc.parallelize(my_data)
kv_write_rdd.saveToRiak(‘kv_sample_bucket’)
Setup
Write](https://image.slidesharecdn.com/3sjohnmusser-161103183222/85/Spark-Summit-EU-talk-by-John-Musser-97-320.jpg)
![Python
import pyspark_riak
conf = pyspark.SparkConf().setAppName("My Spark Riak App")
conf.set("spark.riak.connection.host", "127.0.0.1:8087")
sc = pyspark.SparkContext(conf)
pyspark_riak.riak_context(sc)
my_data = [{'key0':{'data': 0}}, {'key1':{'data': 1}}]
kv_write_rdd = sc.parallelize(my_data)
kv_write_rdd.saveToRiak(‘kv_sample_bucket’)
Setup
Write
Read
kv_read_rdd = sc.riakBucket(‘kv_sample_bucket’).queryAll()
print(kv_read_rdd.collect())](https://image.slidesharecdn.com/3sjohnmusser-161103183222/85/Spark-Summit-EU-talk-by-John-Musser-98-320.jpg)
Spark Summit EU talk by John Musser
- 1. Lessons Learned Optimizing NoSQLfor Apache Spark John Musser @johnmusser / Basho @basho Spark Summit Europe, 2016
- 2. NoSQL
- 5. +NoSQL
- 6. +NoSQL = ?
- 7. + =NoSQL
- 10. + = Webuiltthis.
- 13. Riak • Distributed, key-value NoSQL database • Known for scalability, reliability, ops simplicity • Launched 2009, used by 1/3 of Fortune 50 • Open source (Apache), on GitHub https://github.com/basho/riak/ • Enterprise Edition, see multi-cluster replication
- 14. Westarted to seeinourcustomer base…
- 19. Intellicore Sports Data Platform • 1GB telemetry per driver • 400 packets/second • 1.2M packets/race • Platform setup for 40,000 TPS
- 21. Spark-Riak Connector • Version 1.0: published Sept. 2015 • Current version: 1.6, published Sept. 2016 • Scala / JVM based • Support for Java, Scala, Python • Supports Spark 1.6.x • Open source (Apache), on GitHub https://github.com/basho/spark-riak-connector/
- 23. (this inturn uses learnings from other connectors we’vebuilt...)
- 26. ?
- 53. Properties: "r":"quorum" How Riak Stores Data MiscBucket User123 122883|dave|… Item17Z { “color”:”blue”, “size”:”small”, … } LogoHD Key ValueKeys Values Buckets Bucket Types
- 55. Specify Bucket Load Data val kv_bucket = new Namespace(”MiscBucket") val riakRdd = sc.riakBucket[String](kv_bucket).queryAll() Code: Key/Value Query
- 56. Query by Keys Query by 2i Range val rdd = sc.riakBucket[String](kv_bucket_name) .queryBucketKeys("Alice", "Bob", "Charlie") val rdd = sc.riakBucket[String](kv_bucket_name) .query2iRange("myIndex", 1L, 5000L) Code: Key/Value Query Query by 2i Strings val rdd = sc.riakBucket[String](kv_bucket_name) .query2iKeys("dailyDataIndx", ”Jan", ”Feb”)
- 57. Specify Bucket RDDsLoad Data Key-Value Data Which is “fine”, but, the data is still a bit opaque…
- 61. Specify Bucket Map Schema Key-Value Data Load Data DataFrames Now we have full-fledged DataFrames
- 62. Specify Bucket Map Schema Load Data val kv_bucket = new Namespace(”MiscBucket") case class UserData( user_id: String, name: String, age: Int) val riakRdd = sc.riakBucket[UserData](kv_bucket).queryAll() val df = riakRdd.toDF() Code
- 66. Specify Bucket Map Schema Time Series Data Load Data So let’s use automatic schema discovery instead
- 68. Specify Table Load Data val ts_table_name = "test-table" df = sqlContext.re.option( "spark.riak.connection.hosts", "riak_host_ip:10017") .format("org.apache.spark.sql.riak") .load(ts_table_name) .select(“time”, “col1”, “col2”) .filter(s"time >= CAST($from AS TIMESTAMP)") Time Series Code
- 69. Specify Table Load Data val ts_table_name = "test-table" df = sqlContext.re.option( "spark.riak.connection.hosts", "riak_host_ip:10017") .format("org.apache.spark.sql.riak") .load(ts_table_name) .select(“time”, “col1”, “col2”) .filter(s"time >= CAST($from AS TIMESTAMP)") Time Series Code Use Data df.where(df("age") >= 50).select("id", "name") df.groupBy(”age").count
- 70. Specify Table Load Data val ts_table_name = "test-table" df = sqlContext.re.option( "spark.riak.connection.hosts", "riak_host_ip:10017") .format("org.apache.spark.sql.riak") .load(ts_table_name) .select(“time”, “col1”, “col2”) .filter(s"time >= CAST($from AS TIMESTAMP)") Time Series Code Use Data df.where(df("age") >= 50).select("id", "name") df.groupBy(”age").count Uses the Spark Data Source API
- 73. HTTP Protocol Buffers 2primary interfaces toRiak
- 74. HTTP Protocol Buffers 2primary interfaces toRiak Flexibility Performance
- 75. Protocol Buffers • Data serialization and interchange • Developed by Google • IDL + RPC • Messages serialized to binary wire format • Library support for 20+ languages
- 76. Protocol Buffers • For data serialization and interchange • Originally developed by Google • IDL + RPC • Messages serialized to binary wire format • Library support for 20+ languages Note: In Riak, you typically don’t have to know the details, the client SDKs take care of it for you
- 80. HTTP Protocol Buffers Whatif wecan makethis faster? ?
- 82. HTTP Protocol Buffers Spark-Riak Connector dynamically selects basedon querytype Optimized Binary
- 84. Protocol Buffers BulkTS Operations Optimized Binary Other Operations 30-50% increased throughput
- 90. Riak Time Series SQL Define table CREATE TABLE WEATHER ( region VARCHAR NOT NULL, city VARCHAR NOT NULL, time TIMESTAMP NOT NULL, temperature DOUBLE, PRIMARY KEY( (region, state, QUANTUM(time, 2, 'h')), region, state, time ) )
- 91. Riak Time Series SQL Define table CREATE TABLE WEATHER ( region VARCHAR NOT NULL, city VARCHAR NOT NULL, time TIMESTAMP NOT NULL, temperature DOUBLE, PRIMARY KEY( (region, state, QUANTUM(time, 2, 'h')), region, state, time ) ) The quantum is the tunable key to performance
- 92. Riak Time Series SQL SELECT * FROM WEATHER WHERE city = ’Brussels' time >= ‘2016-01-01’ AND time <= ‘2016-02-01 00:00:00’ Define table Query CREATE TABLE WEATHER ( region VARCHAR NOT NULL, city VARCHAR NOT NULL, time TIMESTAMP NOT NULL, temperature DOUBLE, PRIMARY KEY( (region, state, QUANTUM(time, 2, 'h')), region, state, time ) )
- 93. Be flexible
- 94. Be polyglot
- 96. Python import pyspark_riak conf = pyspark.SparkConf().setAppName("My Spark Riak App") conf.set("spark.riak.connection.host", "127.0.0.1:8087") sc = pyspark.SparkContext(conf) pyspark_riak.riak_context(sc) Setup
- 97. Python import pyspark_riak conf = pyspark.SparkConf().setAppName("My Spark Riak App") conf.set("spark.riak.connection.host", "127.0.0.1:8087") sc = pyspark.SparkContext(conf) pyspark_riak.riak_context(sc) my_data = [{'key0':{'data': 0}}, {'key1':{'data': 1}}] kv_write_rdd = sc.parallelize(my_data) kv_write_rdd.saveToRiak(‘kv_sample_bucket’) Setup Write
- 98. Python import pyspark_riak conf = pyspark.SparkConf().setAppName("My Spark Riak App") conf.set("spark.riak.connection.host", "127.0.0.1:8087") sc = pyspark.SparkContext(conf) pyspark_riak.riak_context(sc) my_data = [{'key0':{'data': 0}}, {'key1':{'data': 1}}] kv_write_rdd = sc.parallelize(my_data) kv_write_rdd.saveToRiak(‘kv_sample_bucket’) Setup Write Read kv_read_rdd = sc.riakBucket(‘kv_sample_bucket’).queryAll() print(kv_read_rdd.collect())
- 99. Spark Streaming
- 100. Spark Streaming
- 101. Setup Stream import com.basho.riak.spark.streaming._ val ssc = new StreamingContext(sparkConf, Seconds(1)) val lines = ssc.socketTextStream(serverIP, serverPort) val errs = lines.filter(lines => lines contains "ERROR") Spark-Riak Streaming Save errs.saveToRiak("test-bucket-4store")
- 103. Deployment
- 104. Deployment
- 106. Simplify
- 110. Don’tdie
- 114. +NoSQL = ?
- 115. + =NoSQL
- 116. + =NoSQL