Building end to end streaming application on Spark
6 likes1,502 views
This document discusses building a real-time streaming application on Spark to analyze sensor data. It describes collecting data from servers through Flume into Kafka and processing it using Spark Streaming to generate analytics stored in Cassandra. The stages involve using files, then Kafka, and finally Cassandra for input/output. Testing streaming applications and redesigning for testability is also covered.
Building end to end streaming application on Spark
- 1. Building End to End Streaming Application on Spark Streaming application development journey https://github.com/Shasidhar/sensoranalytics
- 2. ● Shashidhar E S ● Big data consultant and trainer at datamantra.io ● www.shashidhare.com
- 3. Agenda ● Problem Statement ● Spark streaming ● Stage 1 : File Streams ● Stage 2 : Kafka as input source (Introduction to Kafka) ● Stage 3 : Casandra as Output Store (Introduction to Cassandra) ● Stage 4 : Flume as data collection engine (Introduction to Flume) ● How to test streaming code? ● Next steps
- 4. Earlier System Business model ● Providers of Wi-Fi hot spot devices in public spaces ● Ability to collect data from these devices and analyse Existing System ● Collect data and process in daily batches to generate the required results
- 6. Need for real time engine ● Lot of failures in User login ● Need to analyse why there is a drop in user logins ● Ability to analyse the data in real time rather than daily batches ● As the company is growing Splunk was not scaling as it is not meant for horizontal scaling
- 7. New system requirement ● Able to collect and process large amount of data ● Ability to store results in persistent storage ● A reporting mechanism to view the insights obtained from the analysis ● Need to see the results in real time ● In a simple term, we can call it as a real time monitoring system
- 8. Why Spark Streaming ? ● Easy to port batch system to streaming engine in Spark ● Spark streaming can handle large amounts of data and it is very fast ● Best choice for near real time systems ● Futuristic views ○ Ability to ingest data from many sources ○ Good support for downstream stores like NoSQL ○ And lot more
- 9. Spark Streaming Architecture Server Source directory Spark Streaming engine Output directory View in Zeppelin
- 10. Data format Log Data with the following format ● Timestamp ● Country ● State ● City ● SensorStatus
- 11. Required Results ● Country Wise Stats ○ Hourly,Weekly and Monthly view of total count of records captured countrywise. ● State Wise Stats ○ Hourly,Weekly and Monthly view of total count of records captured statewise. ● City Wise Stats ○ Hourly,Weekly and Monthly view of total count of records captured city wise with respect to sensor status
- 12. Data Analytics - Phase 1 ● Receive data from servers ● Store the input data into files ● Use file as input and output ● Process the data , generate required statistics ● Store results into output files Spark Streaming engine Input files (Directory) Output files (Directory)
- 13. Spark streaming introduction Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams
- 14. Micro batch ● Spark streaming is a fast batch processing system ● Spark streaming collects stream data into small batch and runs batch processing on it ● Batch can be as small as 1s to as big as multiple hours ● Spark job creation and execution overhead is so low it can do all that under a sec ● These batches are called as DStreams
- 15. Apache Zeppelin ● Web based notebook that allows interactive data analysis ● It allows ○ Data ingestion ○ Data Discovery ○ Data Analytics ○ Data Visualization and collaboration ● Built-in Spark integration
- 16. Data Model ● 4 models ○ SensorRecord - To read input records ○ CountryWiseStats - Store country wise aggregations ○ StateWiseStats - Store state wise aggregations ○ CityWiseStats - Store city wise aggregations
- 17. Phase 1 - Hands On Git branch : Master
- 18. Problems with Phase 1 ● Input and output is a file ● Cannot detect new records / new data as and when it is received ● File causes Low latency in system Solution : Replace Input file source with Apache kafka
- 19. Data Analytics - Phase 2 ● Receive data from servers ● Store the input data in Kafka ● Use kafka as input ● Process the data , generate required statistics ● Store results into output files Spark Streaming engine Kafka Output files (Directory)
- 20. Apache Kafka ● High throughput publish subscribe based messaging system ● Distributed, partitioned and replicated commit log ● Messages are persistent in system as Topics ● Uses Zookeeper for cluster management ● Written in scala, but supports many client API’s - Java, Ruby, Python etc ● Developed by LinkedIn
- 22. Terminology ● Topics : Is where messages are maintained and partitioned ● Producers : Processes which produces messages to Topic ● Consumers: Processes which subscribes to topic and read messages ● Brokers: Every server which is part of kafka cluster
- 23. Anatomy of Kafka Topic
- 24. Spark Streaming - Kafka ● Two ways to fetch data from kafka to spark ○ Receiver approach ■ Data is stored in receivers ■ Kafka topic partitions does not correlate with RDDs ■ Enable WAL for zero data loss ■ To increase input speed create multiple receivers
- 25. Spark Streaming - Kafka cont ○ Receiver less approach ■ No data is stored in receivers ■ Exact same partitioning in maintained in Spark RDDs as in Kafka topics ■ No WAL is needed as data is already in kafka we can fetch older data on receiver crash ■ More kafka partitions increases the data fetching speed
- 26. Phase 2 - Hands On Git branch : Kafka
- 27. Problems with Phase 2 ● Output is still a file ● Always full file scan is needed to retrieve, no lookups ● Querying results is cumbersome ● Nosql Database is the better option Solution : Replace Output file with Cassandra
- 28. Data Analytics - Phase 3 Spark Streaming engine Kafka Cassandra ● Receive data from servers ● Store the input data in Kafka ● Use kafka as input ● Process the data , generate required statistics ● Store results into cassandra
- 29. What is Cassandra “Apache Cassandra is an open source, distributed, decentralized, elastically scalable, highly available, fault- tolerant, tunable consistency, column-oriented database” “Daughter of Dynamo and Big Table”
- 30. Key Components and Features ● Distributed ● System keyspace ● Peer to peer - No SPOF ● Read and write to any node ● Operational simplicity ● Gossip and Failure Detection
- 31. Cassandra daemon cassandra (CLI) Language drivers JDBC Drivers Memtable SS tablesCommit Log Overall Architecture
- 32. Spark Cassandra Connector ● Loads data from cassandra to spark and vice versa ● Handles type conversions ● Maps tables to spark RDDs ● Support all cassandra data types, collections and UDTs ● Spark-Sql support ● Supports for Spark SQLs predicate push
- 33. Phase 3 - Hands On Git branch : Cassandra
- 34. Problems with Phase 3 ● Servers cannot push directly to Kafka ● There is an intervention to push data ● Need for automated way to push data Solution : Add Flume as a data collection agent
- 35. Data Analytics - Phase 4 ● Receive data from Server ● Stream data into kafka through flume ● Store the input data in Kafka ● Use kafka as input ● Process the data , generate required statistics ● Store results into cassandra Spark Streaming engine Kafka Cassandra Flume
- 36. Apache Flume ● Distributed data collection service ● Solution for data collection of all formats ● Initially designed to transfer log data into HDFS frequently and reliably ● It is horizontally scalable ● Configurable routing
- 37. Flume Architecture Components ○ Event ○ Source ○ Sink ○ Channel ○ Agent
- 38. Flume Configuration ● Define Source, Sink and Channel names ● Configure Source ● Configure Sink ● Configure Channel ● Bind Source and Sink to Channel
- 39. Phase 4 - Hands On Git branch : Flume
- 40. Data Analytics - Re Design ● Why we want to re design/ re structure ? ● What we want to test ? ● How to test Streaming applications ● Hack a bit on Spark Manual Clock ● Use scala-test for unit testing ● Bring up abstractions to decouple the code ● Write some tests
- 41. Manual Clock ● A clock whose time can be set and modified ● Its notified time will not change as time elapses ● Only callers have control over it ● Specially used for testing
- 42. Phase 5 - Hands On Git branch : unittest
- 43. Next steps ● Use better serialization frameworks like Avro ● Enable Checkpointing ● Integrate kafka monitoring tools ● Adding support for multiple kafka topics ● Write more tests for all functionality