Scylla Summit 2018: Best Practices for Running Spark with Scylla
Download as PPTX, PDF1 like1,781 views
The document discusses best practices for integrating Apache Spark with Scylla, highlighting the role of Eyal Gutkind, the head of solutions architects at Scylla. It covers the architecture of Scylla tokens, data partitioning considerations, and the use of the Cassandra-Spark connector for managing data interactions. Key performance tuning tips for running Spark with Scylla are provided, emphasizing resource management and Spark configuration settings.
Scylla Summit 2018: Best Practices for Running Spark with Scylla
- 1. Best Practices for Running Spark with Scylla Eyal Gutkind - Head of Solutions Architects
- 2. Eyal Gutkind is head of solution architects at Scylla. Prior to Scylla Eyal held product management roles at Mirantis and DataStax. Prior to DataStax Eyal spent 12 years with Mellanox Technologies in various engineering management and product marketing roles. Eyal holds a BSc. degree in Electrical and Computer Engineering from Ben Gurion University, Israel and MBA from Fuqua School of Business at Duke University, North Carolina. Speaker
- 3. Analytics
- 4. Scylla token architecture source: http://docs.scylladb.com/architecture/ringarchitecture/
- 5. Scylla token architecture source: http://docs.scylladb.com/architecture/ringarchitecture/
- 6. Spark and Spark partitions source: https://spark.apache.org/docs/latest/cluster-overview.html
- 7. Spark and Spark partitions Node 1 RDD1 Partition 1 RDD2 Partition 4 Node 2 RDD1 Partition 4 RDD2 Partition 2 Node 3 RDD1 Partition 2 RDD2 Partition 3 Node 4 RDD1 Partition 3 RDD2 Partition 1
- 8. 8 Scylla to Spark, partition considerations RDD 1 Partition 3 Pkey1 Col1 Col2 Col3 Col1 Col2 Col3Pkey2 Col1 Col2 Col3Pkey7342
- 9. The Cassandra-Spark connector https://github.com/datastax/spark-cassandra-connector âȘ Provides Spark Context to data stored in Scylla/Cassandra âȘ Batch writes âȘ Read Scylla/Cassandra partitions to Spark Partitions âȘ Connection management between Scylla and Spark driver and executors âȘ Utilizes the Cassandra Java driver
- 10. When Spark writes to Scylla 10 output.batch.grouping.buffer.size output.batch.size.bytes output.concurrent.writes output.batch.grouping.key
- 11. When Spark reads from Scylla 11 input.split.size_in_mb Donât forget data is compressed on Disk! Scylla paging capabilities will have an impact! input.fetch.size_in_rows
- 12. To collocate or not to collocate?
- 13. âȘ Increase default Spark parallelism (number of cores in the Spark local machine deployment) âȘ Reduced Spark split size (64 -> 1) âȘ Connection.connections_per_executor_max (# of core or more) âȘ Output.concurrent.writes default 5 âȘ Concurrent.reads default is 512 Fine tuning Spark performance with Scylla
- 14. âȘ Scylla enables analytics on top transactional data âȘ Performance tuning is required for certain workloads âȘ Resource management is key to stability of your deployment Conclusion
- 15. Q&A Stay in touch Learn more eyal@scylladb.com @gutkinde scylladb.com/blog scylladb-users.slack.com