Spark Powered by Scylla
3 likes2,024 views
Eyal Gutkind is the Head of Solutions Architects at Scylla, with prior experience in product management roles at Mirantis and DataStax, and engineering management at Mellanox Technologies. Scylla is a next-generation NoSQL database that serves as a drop-in replacement for Cassandra, offering significantly improved performance and low latency. The document also discusses integration with Spark, highlighting the importance of performance tuning and resource management for optimal deployment stability.
Spark Powered by Scylla
- 1. Eyal Gutkind - Head of Solutions Architects WEBINAR
- 2. 2 Eyal Gutkind is head of solution architects at Scylla. Prior to Scylla Eyal held product management roles at Mirantis and DataStax. Prior to DataStax Eyal spent 12 years with Mellanox Technologies in various engineering management and product marketing roles. Eyal holds a BSc. degree in Electrical and Computer Engineering from Ben Gurion University, Israel and MBA from Fuqua School of Business at Duke University, North Carolina.
- 3. 3 + Next-generation NoSQL database + Drop-in replacement for Cassandra + 10X the performance & low tail latency + Open source and enterprise editions + Founded by the creators of KVM hypervisor + HQs: Palo Alto, CA; Herzelia, Israel
- 4. 4
- 5. 5
- 6. 6
- 7. 7
- 8. 8
- 9. 9
- 10. 10
- 11. 11
- 12. 12 https://github.com/datastax/spark-cassandra-connector + Provides Spark Context to data stored in Scylla/Cassandra + Batch writes + Read Scylla/Cassandra partitions to Spark Partitions + Connection management between Scylla and Spark driver and executors + Utilizes the Cassandra Java driver
- 14. 14 input.split.size_in_mb Don’t forget data is compressed on Disk! Scylla paging capabilities will have an impact! input.fetch.size_in_rows
- 15. 15
- 16. 16
- 17. 17
- 18. 18
- 19. 19
- 20. + ncrease default Spark parallelism (number of cores in the Spark local machine deployment) + Reduced Spark split size (64 -> 1) + Connection.connections_per_executor_max (# of core or more) + Output.concurrent.writes default 5 + Concurrent.reads default is 512 20
- 21. 21
- 22. 22
- 23. 23
- 24. + Scylla enables analytics on top transactional data + Performance tuning is required for certain workloads + Resource management is key to stability of your deployment 24
- 25. - Docker based deployment example: - https://www.scylladb.com/2018/06/04/mms-day-13-spark-hive/ - Scylla and Spark integration example: - http://docs.scylladb.com/kb/scylla-and-spark-integration/ - Cassandra-Spark connector: - https://github.com/datastax/spark-cassandra-connector - Good source of information about Cassandra-Spark connector - http://www.russellspitzer.com/ - 25
- 26. 26
- 27. United States Israel www.scylladb.com @scylladb