Real-Time Analytics with Apache Cassandra and Apache Spark
32 likes9,157 views
This document provides an overview of real-time analytics with Apache Cassandra and Apache Spark. It discusses how Spark can be used for stream processing over Cassandra for storage. Spark Streaming ingests real-time data from sources like Kafka and processes it using Spark transformations and actions. The processed data can be stored in Cassandra for querying. Cassandra is well suited for high write throughput and storing large amounts of data, while Spark enables fast in-memory processing and machine learning capabilities. Together, Spark and Cassandra provide a scalable solution for real-time analytics and querying of large datasets.
![Apache Cassandra â More than one server
All nodes participate in a cluster
Shared nothing
Add or remove as needed
More capacity? Add more servers
Node is a basic unit inside a cluster
Each node owns a range of partitions
Consistent Hashing
Node 1
Node 2
Node 3
Node 4
[26-50]
[0-25]
[51-75]
[76-100] [0-25]
[0-25]
[26-50]
[26-50]
[51-75]
[51-75]
[76-100]
[76-100]](https://image.slidesharecdn.com/real-time-analytics-with-spark-and-cassandra-v1-151125191956-lva1-app6892/85/Real-Time-Analytics-with-Apache-Cassandra-and-Apache-Spark-40-320.jpg)
Real-Time Analytics with Apache Cassandra and Apache Spark
- 1. BĂLE BERNE BRUGG DUSSELDORF FRANCFORT S.M. FRIBOURG E.BR. GENĂVE HAMBOURG COPENHAGUE LAUSANNE MUNICH STUTTGART VIENNE ZURICH Real-Time Analytics with Apache Cassandra and Apache Spark Guido Schmutz
- 2. Guido Schmutz âą Working for Trivadis for more than 18 years âą Oracle ACE Director for Fusion Middleware and SOA âą Author of different books âą Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data âą Technology Manager @ Trivadis âą More than 25 years of software development experience âą Contact: guido.schmutz@trivadis.com âą Blog: http://guidoschmutz.wordpress.com âą Twitter: gschmutz
- 3. Agenda 1. Introduction 2. Apache Spark 3. Apache Cassandra 4. Combining Spark & Cassandra 5. Summary
- 4. Big Data Definition (4 Vs) + Time to action ? â Big Data + Real-Time = Stream Processing Characteristics of Big Data: Its Volume, Velocity and Variety in combination
- 5. What is Real-Time Analytics? What is it? Why do we need it? How does it work? ⹠Collect real-time data ⹠Process data as it flows in ⹠Data in Motion over Data at Rest ⹠Reports and Dashboard access processed data Time Events RespondAnalyze Short time to analyze & respond § Required - for new business models § Desired - for competitive advantage
- 6. Real Time Analytics Use Cases âą Algorithmic Trading âą Online Fraud Detection âą Geo Fencing âą Proximity/Location Tracking âą Intrusion detection systems âą Traffic Management âą Recommendations âą Churn detection âą Internet of Things (IoT) / Intelligence Sensors âą Social Media/Data Analytics âą Gaming Data Feed âą âŠ
- 7. Apache Spark
- 8. Motivation â Why Apache Spark? Hadoop MapReduce: Data Sharing on Disk Spark: Speed up processing by using Memory instead of Disks map reduce . . . Input HDFS read HDFS write HDFS read HDFS write op1 op2 . . . Input Output Output
- 9. Apache Spark Apache Spark is a fast and general engine for large-scale data processing âą The hot trend in Big Data! âą Originally developed 2009 in UC Berkleyâs AMPLab âą Based on 2007 Microsoft Dryad paper âą Written in Scala, supports Java, Python, SQL and R âą Can run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk âą One of the largest OSS communities in big data with over 200 contributors in 50+ organizations âą Open Sourced in 2010 â since 2014 part of Apache Software foundation
- 10. Apache Spark Spark SQL (Batch Processing) Blink DB (Approximate Querying) Spark Streaming (Real-Time) MLlib, Spark R (Machine Learning) GraphX (Graph Processing) Spark Core API and Execution Model Spark Standalone MESOS YARN HDFS Elastic Search NoSQL S3 Libraries Core Runtime Cluster Resource Managers Data Stores
- 11. Resilient Distributed Dataset (RDD) Are âą Immutable âą Re-computable âą Fault tolerant âą Reusable Have Transformations âą Produce new RDD âą Rich set of transformation available âą filter(), flatMap(), map(), distinct(), groupBy(), union(), join(), sortByKey(), reduceByKey(), subtract(), ... Have Actions âą Start cluster computing operations âą Rich set of action available âą collect(), count(), fold(), reduce(), count(), âŠ
- 12. RDD RDD Input Source âą File âą Database âą Stream âą Collection .count() -> 100 Data
- 16. Stage 1 â reduceByKey() Stage 1 â flatMap() + map() Spark Workflow Input HDFS File HadoopRDD MappedRDD ShuffledRDD Text File Output sc.hapoopFile() map() reduceByKey() sc.saveAsTextFile() Transformations (Lazy) Action (Execute Transformations) Master MappedRDD P0 P1 P3 ShuffledRDD P0 MappedRDD flatMap() DAG Scheduler
- 17. Spark Workflow HDFS File Input 1 HadoopRDD FilteredRDD MappedRDD ShuffledRDD HDFS File Output HadoopRDD MappedRDD HDFS File Input 2 SparkContext.hadoopFile() SparkContext.hadoopFile()filter() map() map() join() SparkContext.saveAsHadoopFile() Transformations (Lazy) Action (Execute Transformations)
- 19. Stage 1 â flatMap() + map() Spark Execution Model Data Storage Worker Master Executer Data Storage Worker Executer Data Storage Worker Executer RDD P0 P1 P3 Narrow TransformationMaster filter() map() sample() flatMap() Data Storage Worker Executer
- 20. Stage 2 â reduceByKey() Spark Execution Model Data Storage Worker Executer Data Storage Worker Executer RDD P0 Wide Transformation Master join() reduceByKey() union() groupByKey() Shuffle ! Data Storage Worker Executer Data Storage Worker Executer
- 21. Batch vs. Real-Time Processing Petabytes of Data Gigabytes Per Second
- 23. Apache Kafka distributed publish-subscribe messaging system Designed for processing of real time activity stream data (logs, metrics collections, social media streams, âŠ) Initially developed at LinkedIn, now part of Apache Does not use JMS API and standards Kafka maintains feeds of messages in topics Kafka Cluster Consumer Consumer Consumer Producer Producer Producer
- 24. Apache Kafka Kafka Broker Temperature Processor Temperature Topic Rainfall Topic 1 2 3 4 5 6 Rainfall Processor1 2 3 4 5 6 Weather Station
- 25. Apache Kafka Kafka Broker Temperature Processor Temperature Topic Rainfall Topic 1 2 3 4 5 6 Rainfall Processor Partition 0 1 2 3 4 5 6 Partition 0 1 2 3 4 5 6 Partition 1 Temperature Processor Weather Station
- 26. Apache Kafka Kafka Broker Temperature Processor Weather Station Temperature Topic Rainfall Topic Rainfall Processor P 0 Temperature Processor 1 2 3 4 5 P 1 1 2 3 4 5 Kafka Broker Temperature Topic Rainfall Topic P 0 1 2 3 4 5 P 1 1 2 3 4 5 P 0 1 2 3 4 5 P 0 1 2 3 4 5
- 30. Discretized Stream (DStream) Kafka Weather Station Weather Station Weather Station Discrete by time Individual Event DStream = RDD
- 31. Discretized Stream (DStream) DStream DStream X Seconds Transform .countByValue() .reduceByKey() .join .map
- 32. Discretized Stream (DStream) time 1 time 2 time 3 message time nâŠ. f(message 1) RDD @time 1 f(message 2) f(message n) âŠ. message 1 RDD @time 1 message 2 message n âŠ. result 1 result 2 result n âŠ. message message message f(message 1) RDD @time 2 f(message 2) f(message n) âŠ. message 1 RDD @time 2 message 2 message n âŠ. result 1 result 2 result n âŠ. f(message 1) RDD @time 3 f(message 2) f(message n) âŠ. message 1 RDD @time 3 message 2 message n âŠ. result 1 result 2 result n âŠ. f(message 1) RDD @time n f(message 2) f(message n) âŠ. message 1 RDD @time n message 2 message n âŠ. result 1 result 2 result n âŠ. Input Stream Event DStream MappedDStream map() saveAsHadoopFiles() Time Increasing DStreamTransformation Lineage Actions Trigger Spark Jobs Adapted from Chris Fregly: http://slidesha.re/11PP7FV
- 33. Apache Spark Streaming â Core concepts Discretized Stream (DStream) âą Core Spark Streaming abstraction âą micro batches of RDDâs âą Operations similar to RDD Input DStreams âą Represents the stream of raw data received from streaming sources âą Data can be ingested from many sources: Kafka, Kinesis, Flume, Twitter, ZeroMQ, TCP Socket, Akka actors, etc. âą Custom Sources can be easily written for custom data sources Operations âą Same as Spark Core + Additional Stateful transformations (window, reduceByWindow)
- 34. Apache Cassandra
- 35. Apache Cassandra Apache Cassandra⹠is a free ⹠Distributed⊠⹠High performance⊠⹠Extremely scalable⊠⹠Fault tolerant (i.e. no single point of failure)⊠post-relational database solution Optimized for high write throughput
- 36. Apache Cassandra - History Bigtable Dynamo
- 37. Motivation - Why NoSQL Databases? aaa âą Dynamo Paper (2007) âą How to build a data store that is âą Reliable âą Performant âą âAlways Onâ âą Nothing new and shiny âą 24 other papers cited âą Evolutionary
- 38. Motivation - Why NoSQL Databases? âą Google Big Table (2006) âą Richer data model âą 1 key and lotâs of values âą Fast sequential access âą 38 other papers cited
- 39. Motivation - Why NoSQL Databases? âą Cassandra Paper (2008) âą Distributed features of Dynamo âą Data Model and storage from BigTable âą February 2010 graduated to a top-level Apache Project
- 40. Apache Cassandra â More than one server All nodes participate in a cluster Shared nothing Add or remove as needed More capacity? Add more servers Node is a basic unit inside a cluster Each node owns a range of partitions Consistent Hashing Node 1 Node 2 Node 3 Node 4 [26-50] [0-25] [51-75] [76-100] [0-25] [0-25] [26-50] [26-50] [51-75] [51-75] [76-100] [76-100]
- 41. Apache Cassandra â Fully Replicated Client writes local Data syncs across WAN Replication per Data Center Node 1 Node 2 Node 3 Node 4 Node 1 Node 2 Node 3 Node 4 West East Client
- 42. Apache Cassandra What is Cassandra NOT? âą A Data Ocean âą A Data Lake âą A Data Pond âą An In-Memory Database âą A Key-Value Store âą Not for Data Warehousing What are good use cases? âą Product Catalog / Playlists âą Personalization (Ads, Recommendations) âą Fraud Detection âą Time Series (Finance, Smart Meter) âą IoT / Sensor Data âą Graph / Network data
- 43. How Cassandra stores data ⹠Model brought from Google Bigtable ⹠Row Key and a lot of columns ⹠Column names sorted (UTF8, Int, Timestamp, etc.) Column Name ⊠Column Name Column Value Column Value Timestamp Timestamp TTL TTL Row Key 1 2 Billion Billion of Rows
- 44. Combining Spark & Cassandra
- 45. Spark and Cassandra Architecture â Great Combo Good at analyzing a huge amount of data Good at storing a huge amount of data
- 46. Spark and Cassandra Architecture Spark Streaming (Near Real-Time) SparkSQL (Structured Data) MLlib (Machine Learning) GraphX (Graph Analysis)
- 47. Spark and Cassandra Architecture Spark Connector Weather Station Spark Streaming (Near Real-Time) SparkSQL (Structured Data) MLlib (Machine Learning) GraphX (Graph Analysis) Weather Station Weather Station Weather Station Weather Station
- 48. Spark and Cassandra Architecture âą Single Node running Cassandra âą Spark Worker is really small âą Spark Master lives outside a node âą Spark Worker starts Spark Executer in separate JVM âą Node local Worker Master Executer Executer Server Executer
- 49. Spark and Cassandra Architecture Worker Worker Worker Master Worker âą Each node runs Spark and Cassandra âą Spark Master can make decisions based on Token Ranges âą Spark likes to work on small partitions of data across a large cluster âą Cassandra likes to spread out data in a large cluster 0-25 26-50 51-75 76-100 Will only have to analyze 25% of data!
- 50. Spark and Cassandra Architecture Master 0-25 26-50 51-75 76-100 Worker Worker WorkerWorker 0-25 26-50 51-75 76-100 Transactional Analytics
- 51. Cassandra and Spark Cassandra Cassandra & Spark Joins and Unions No Yes Transformations Limited Yes Outside Data Integration No Yes Aggregations Limited Yes
- 52. Summary
- 53. Summary Kafka âą Topics store information broken into partitions âą Brokers store partitions âą Partitions are replicated for data resilience Cassandra âą Goals of Apache Cassandra are all about staying online and performant âą Best for applications close to your users âą Partitions are similar data grouped by a partition key Spark âą Replacement for Hadoop Map Reduce âą In memory âą More operations than just Map and Reduce âą Makes data analysis easier âą Spark Streaming can take a variety of sources Spark + Cassandra âą Cassandra acts as the storage layer for Spark âą Deploy in a mixed cluster configuration âą Spark executors access Cassandra using the DataStax connector
- 54. Lambda Architecture with Spark/Cassandra Data Collection (Analytical) Batch Data Processing Batch compute Result StoreData Sources Channel Data Access Reports Service Analytic Tools Alerting Tools Social (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir)
- 55. Lambda Architecture with Spark/Cassandra Data Collection (Analytical) Batch Data Processing Batch compute Result StoreData Sources Channel Data Access Reports Service Analytic Tools Alerting Tools Social (Analytical) Real-Time Data Processing Stream/Event Processing Batch compute Messaging Result Store Query Engine Result Store Computed Information Raw Data (Reservoir)