Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka

Editor's Notes

• #2: Thank you for coming to hear about our different use-cases of Streaming with Spark and Kafka I will try to make it interesting and valuable for you
• #3: Questions - at the end of the session
• #6: Nielsen marketing cloud or NMC in short A group inside Nielsen, Born from exelate company that was acquired by Nielsen on March 2015 Nielsen is a data company and so are we and we had strong business relationship until at some point they decided to go for it and acquired exelate Data company meaning Buying and onboarding data into NMC from data providers, customers and Nielsen data We have huge high quality dataset enrich the data using machine learning models in order to create more relevant quality insights categorize and sell according to a need Helping brands to take intelligence business decisions E.g. Targeting in the digital marketing world Meaning help fit ads to viewers For example street sign can fit to a very small % of people who see it vs Online ads that can fit the profile of the individual that sees it More interesting to the user More chances he will click the ad Better ROI for the marketer
• #7: What are the questions we try to answer in NMC that help our customers to take business decisions ? A lot of questions but to lead to what druid is coming to solve Translating from human problem to technical problem: UU (distinct) count Simple count
• #8: Few words on NMC data pipeline architecture: Frontend layer: Receives all the online and offline data traffic Bare metal on different data centers (3 in US, 2 in EU ,3 in APAC) near real time - high throughput/low latency challenges Backend layer Aws Cloud based process all the frontend layer outputs ETL’s - load data to data sources aggregated and raw Applications layer Also in the cloud Variety of apps above all our data sources Web - NMC data configurations (segments, audiences etc) campaign analysis , campaign management tools etc. visualized profile graphs reports
• #9: We’ve used Clustrix (our operation DB) as both OLTP and OLAP Events are flowing from our Serving system, need to ETL the data into our data stores (DB, DWH, etc.) Events were written to CSV files Some fields had double quotes, e.g: 2014-07-17,12:55:38,2,2,0,"1619691,9995",1 Processing was done via standalone Java process Had many problems with this architecture Truncated lines in input files Can’t enforce schema Had to “manually” scale the processes
• #11: Around 2014 the standalone Java processes were transformed into Spark batch jobs written in Scala (but in this presentation we’re going to focus on streaming). This is a simplified version of what we built (simplified it to make it clearer across the presentation) Spark A distributed, scalable engine for large-scale data processing Unified framework for batch, streaming, machine learning, etc Was gaining a lot of popularity in the Big Data community Built on RDDs (Resilient distributed dataset) A fault-tolerant collection of elements that can be operated on in parallel Scala Combines object-oriented and functional programming First-class citizen is Spark
• #13: Kafka Open-source stream-processing platform Highly scalable Publish/Subscribe (A.K.A pub/sub) Schema enforcement - using Schema Registry and relying on Avro format Much more Originally developed by LinkedIn Graduated from Apache Incubator on late 2012 Quickly became the de facto standard in the industry Today commercial development is led by Confluent Spark Streaming A natural evolvement of our Spark batch job (unified framework – remember?) Introduced the DStream concept Continuous stream of data Represented by a continuous series of RDDs Works in micro-batches Each RDD in a DStream contains data from a certain interval (e.g 5 minutes)
• #14: We started with Spark Streaming over Kafka (in 2015) Our Streaming apps were “stateless” (see below) and running 24/7 : Reading a batch of messages from Kafka Performing simple transformations on each message (no aggregations) Writing the output of each batch to a persistent storage (DB, S3, etc.) Stateful operations (aggregations) were performed periodically in batch either by Spark jobs ETLs in our DB/DWH
• #16: Looking back, Spark Streaming might have been able to perform stateful operations for us, but (as far as I recall) mapWithState wasn’t available yet, and updateStateByKey had some pending issues. The way to achieve it was : Read messages from Kafka Aggregate the messages of the current micro-batch Increased micro-batch length to achieve a better aggregation ratio Combine the results of the results of the previous micro-batches (stored on the cluster’s HDFS) Write the results back to HDFS Every X batches : Update the DB with the aggregated data (some sort of UPSERT) Delete the aggregated files from HDFS UPSERT = INSERT ... ON DUPLICATE KEY UPDATE … (in MySQL) For example, given t1 with columns a (the key) and b (starting from an empty table) INSERT INTO t1 (a,b) VALUES (1,2) ON DUPLICATE KEY UPDATE b=b+VALUES(b); -> a=1, b=2 INSERT INTO t1 (a,b) VALUES (1,5) ON DUPLICATE KEY UPDATE b=b+VALUES(b); -> a=1, b=7
• #17: In this specific use-case, the app was reading from a topic which had only small amounts of data Required us to manage the state on our own Error-prone E.g what if my cluster is terminated and data on HDFS is lost? Complicates the code Mixed input sources for the same app (Kafka + files) Possible performance impact Might cause the Kafka consumer to lag Obviously not the perfect way (but that’s what we had…)
• #18: DataFrame/Dataset - rather than DStream’s RDD Catalyst Optimizer - extensible query optimizer which is “at the core of Spark SQL… designed with these key two purposes: Easily add new optimization techniques and features to Spark SQL Enable external developers to extend the optimizer (e.g. adding data source specific rules, support for new data types, etc.)” (see https://databricks.com/glossary/catalyst-optimizer) Other features included : Handling event-time and late data End-to-end exactly-once fault-tolerance
• #19: Checkpoint folder is the location Spark stores : The offsets we already read from Kafka The state of the stateful operations (e.g aggregations) We’ve used S3 (via EMRFS) for checkpointing. We’ve deployed to production various use-cases using Structured Streaming : Periodic Spark batch job was converted to a stateless Structured Streaming app Periodic standalone Java app was converted to a stateful Structured Streaming app A brand new app was written as a stateful Structured Streaming app
• #20: EMRFS consistent view - an optional feature on AWS EMR, allows clusters to check for list and read-after-write consistency for S3 objects written by or synced with EMRFS Checkpointing to S3 wasn’t straight-forward Try using EMRFS consistent view Recommended for stateless apps For stateful apps, we encountered sporadic issues possibly related to the metadata store (i.e DynamoDB)
• #21: Strengths : Running incremental, continuous processing End-to-end exactly-once fault-tolerance (if you implement it correctly) Increased performance (uses the Catalyst SQL optimizer and other DataFrame optimizations like code generation) Massive efforts are invested in it Weaknesses : Maturity Inability to perform multiple actions on the exact same Dataset E.g http://apache-spark-user-list.1001560.n3.nabble.com/Structured-Streaming-Avoiding-multiple-streaming-queries-tt30944.html Seems to be resolved by https://issues.apache.org/jira/browse/SPARK-24565 (in Spark 2.4, but then you get at-least once)
• #22: Moved many apps (mostly the ones performing UU counts and “hit” counts) to rely on Druid, which is meant for OLAP, so now : Spark Streaming app Runs on a long-lived EMR cluster (cluster is on 24/7) Performs the “in-batch” aggregation per micro-batch (before writing to S3) Writes relevant metadata to RDS (e.g S3 path) This kind of “split” (i.e persisting the Dataset/DataFrame and iterating it a few times) is impossible with Structured Streaming (where every “branch” of processing is a separate query, at least until https://issues.apache.org/jira/browse/SPARK-24565) M/R ingestion job (loads data into Druid) : Reads relevant metadata from RDS performs the final aggregation (before data is loaded into Druid) Update state in RDS (e.g which files were handled)
• #23: Screenshot from Ganglia installed on our AWS EMR cluster running the Spark Streaming app Remember - longer micro-batches result in a better aggregation ratio Each such app runs on its own long-lived EMR cluster
• #24: Extreme load of Kafka brokers’ disks Each micro-batch needs to read ~300M messages , Kafka can’t store it all in memory ConcurrentModificationException when using Spark Streaming + Kafka 0.10 integration Forced us to use 1 core per executor to avoid it https://issues.apache.org/jira/browse/SPARK-19185 supposedly solved in 2.4.0 (possibly solving https://issues.apache.org/jira/browse/SPARK-22562 as well) We wish we could run it even less frequently Remember - longer micro-batches result in a better aggregation ratio
• #25: Each Kafka topic has its own RDR loader, which stores the data in a separate bucket on S3 (partitioned by date) This means each topic has only 1 consumer (the appropriate RDR loader) RDR loaders use micro-batches of 4-6 minutes, writing about 100 files per micro-batch, each file is ~0.5GB (to allow efficient read) Only simple transformations on each message (no aggregations) Hence no need for long micro-batche
• #26: Once the RDR loader writes the files to S3, it also writes the files’ paths to a designated topic in Kafka
• #27: How does that work? Spark batch applications are executed every X hours On each execution : A transient EMR cluster is launched An app consumes the next Y messages from the designated Kafka topic (containing Y paths).Since each such app is consuming from Kafka is a standard way, the offsets (i.e which messages the consumer already read) are committed (and maintained) the same way as we do for any other Kafka consumer Then the app reads those Y paths from RDR and processes them Once done, the EMR cluster is terminated
• #28: Applications are now batch rather than streaming We no longer use Spark Streaming-Kafka integration, but rather Kafka API (from the driver) to read the files’ paths from the designated Kafka topic Once we got the paths from Kafka, we use the “regular” batch method of reading files, i.e spark.read.parquet After processing has ended, offsets of the messages we read are committed (as we’d do for any Kafka consumer) We now use Airflow (the de facto standard in the industry) to schedule and monitor our batch jobs All this obviously is not meant to be used by apps that require actual real time (say milliseconds)
• #29: EMR clusters are now transient, so the cluster is terminated as soon as the batch job has finished - no more idle clusters Cost: Spark Streaming cluster is on 24/7, so the cost is fixed With the new infra, the daily cost varies based on the amount of data we processed that day
• #31: Initially replaced standalone Java with Spark & Scala Solved the scale-related issues but not the CSV-related issues Introduced Spark Streaming & Kafka for “stateless” use-cases Replaced CSV files with Kafka (de facto standard in the industry) Already had Spark batch in production (Spark as a unified framework) Tried Spark Streaming for stateleful use-cases (via “local” aggregations) Not the optimal solution Moved to Structured Streaming (for all use-cases) Pros include : Enables running continuous, incremental processes Built on Spark SQL Cons include : Maturity Inability to perform multiple actions on the exact same Dataset
• #32: Went back to Spark Streaming Aggregations are done per micro-batch (in Spark) and daily (in Druid) Still not perfect Performance penalty in Kafka for long micro-batches Concurrency issue with Kafka 0.10 consumer in Spark Under-utilized Spark clusters Introduced “streaming” over our Data Lake Spark Streaming apps (A.K.A “RDR loaders”) write files to S3 and paths to Kafka Spark batch apps read S3 paths from Kafka (and the actual files from S3) Transient EMR clusters Airflow for scheduling and monitoring Pros : Eliminated the performance penalty we had in Kafka Spark clusters are much better utilized = $$$ saved
• #37: “The key idea in Structured Streaming is to treat a live data stream as a table that is being continuously appended… You will express your streaming computation as standard batch-like query as on a static table, and Spark runs it as an incremental query on the unbounded input table”
• #39: “A query on the input will generate the “Result Table”. Every trigger interval (say, every 1 second), new rows get appended to the Input Table, which eventually updates the Result Table. Whenever the result table gets updated, we would want to write the changed result rows to an external sink.”