Real-time Data Pipeline: Kafka Streams / Kafka Connect versus Spark Streaming
3 likes3,739 views
This document compares Apache Spark Streaming and Apache Kafka for real-time data pipelines. It outlines the key differences between the two in areas like new file detection, processing, failure handling, deployment, scaling, and monitoring. Some key points are that Spark Streaming allows detecting new files within directories but requires separate streams for different data sources, while Kafka can detect new files across sources using a watcher connector. Kafka Connect is also better for scaling tasks up and down dynamically compared to Spark Streaming. The document recommends considering your specific data sources, sinks, and integration testing needs to determine the best solution.
Real-time Data Pipeline: Kafka Streams / Kafka Connect versus Spark Streaming
- 1. REX Real-time Data Pipeline: Kafka Streams / Kafka Connect versus Spark Streaming EL ARIB Abdelhamide , Data Engineer at
- 2. OUTLINE • The Challenge • Brief presentation of Spark & Kafka • Kafka vs Spark : 1. New File Detection 2. Processing 3. Handling Failure 4. Deployment 5. Scaling 6. Monitoring • 10 points not to forget
- 3. THE CHALLENGE HDFS is our source of truth Need to stream very big data (parquet or Avro) into Kafka Different schemas for each source Need to scale up/down the speed of streaming Exactly-Once Copy (Hdfs & Kafka) And the monitoring is, as always, a must HDFS /client1/crm /client1/tracking /client1/transactions /client2/mainCrm /client2/payments ….
- 4. BRIEF PRESENTATION OF SPARK & KAFKA
- 5. APACHE SPARK Distributed, data processing, in-memory engine for batch & streaming mode. Apache Spark Core Standalone Scheduler MesosYarn k8s …Cluster Manager Spark SQL Spark Streaming Spark MLlibGraphXLibraries
- 6. SPARK STREAMING Receiver Records Micro Batches (RDDs) Apache Spark Core Standalone Scheduler MesosYarn k8s … End-to-end latency ~ 100 ms Exactly once guarantee
- 7. SPARK STRUCTURED STREAMING Whenever you can, use Dataframe instead of Spark’s RDD primitive DataFrames get the benefit of the Catalyst query optimizer. Multiple types of Triggers Receiver Records Micro Batches (DataFrames) Apache Spark Core Standalone Scheduler MesosYarn k8s … (Experimental in Spark 2.3) End to end latency ~ 1 ms with at least once guarantee Spark SQL (Catalyst)
- 8. KAFKA/ KAFKA STREAMS/ KAFKA CONNECT Data Source Kafka Connect Kafka Topic Kafka Streams
- 9. KAFKA CONNECT Kafka Connect is a framework to stream data into & out of Kafka using connectors. Two types of connectors: Sinks (Export) & Sources (Import) poll.interval.ms (default 5ms)
- 10. Kafka Connect Cluster Worker 1 Worker 2 Worker 3 Connector Elastic Sink Instance Conn ES, Task 1 Partitions: 1,2 Conn Elastic Sink, Task 2 Partitions: 3,4 Conn Elastic Sink, Task 3 Partitions: 5,6
- 11. 1. NEW FILE DETECTION
- 12. 1. NEW FILE DETECTION - SPARK Can detect new files within a dir Should have the same schema Can’t create one stream for all source of the client X N clients with M Source => N*M streams. 1 executor at least for each stream HDFS /client1/crm /client1/tracking /client1/transactions …
- 13. 1. NEW FILE DETECTION – KAFKA
- 14. 1. NEW FILE DETECTION - HDFS-WATCHER (INOTIFY) /client1/crm /client1/tracking /client1/transactions /client2/mainCrm /client2/payments …. HDFS-WATCHER HDFS Kafka Topic (Avro) (source, client) -> path Scanning for new files (exclude dirs with _temporary)
- 15. 2. PROCESSING
- 16. 2. PROCESSING - SPARK Read file as stream Read partition by partition At least once when writing to Kafka
- 17. 2. PROCESSING - KAFKA HDFS Task 1 Kafka Connect Kafka Producer Client_source Topic hdfs-source- offset Topic GlobalKtable (path, lastLineCommitted) HDFS Watcher Schema Registry poll()commitRecords() (path, newLastLineCommitted) FileDetection Kafka topic ((source, client) -> filePath) Get n (=batch.size) lines Consuming new files to process & get keyPartitionKey from the metadata in their schemas Kafka Connect Framework Tracking the offset so we can handle the failure 1 2 3 5 6 7 8 4
- 19. 3. HANDLING FAILURE - SPARK The HDFS receiver is not reliable receiver .option("checkpointLocation", ”hdfs://checkpointPath") (Expensive) spark.streaming.receiver.writeAheadLog.enable spark.task.maxFailures = 0 trait StreamingListener { … /** Called when a receiver has reported an error */ def onReceiverError(receiverError: StreamingListenerReceiverError) { } … }
- 20. 3. HANDLING FAILURE - KAFKA GlobalKtable that track the last line commited. KIP-298: Error Handling in Connect • errors.retry.timeout • errors.retry.delay.max.ms • errors.tolerance (in a task) = none Kafka producer config: • acks = 1 (ou all) • Retries > 0 • max.in.flight.requests.per.connection > 0( no need to preserve order) • Batch.size (producer config) < number of lines * size of a single line • linger.ms = 0 (Send the record as they arrive)
- 21. 4. DEPLOYMENT
- 22. 4. DEPLOYMENT - SPARK Standalone Cluster mode: Using spark-submit, the driver is in the cluster Client mode : Start the app. The driver is in the Client machine.
- 23. 4. DEPLOYMENT - KAFKA Packager the connector (Classpath issues warning) Make it in /plugins Start Kafka connect cluster Run via the The rest API
- 25. 5. SCALING UP/DOWN- SPARK Dynamic Allocation Min*, max & init resources Can’t scale down manually the resources for executor without stopping the driver. Can’t scale up/down the number of executors manually.
- 26. 5. SCALING UP/DOWN- KAFKA Scale up/down easily the number of task. Only Static allocation for the cluster resources. Can’t update the cluster/workers resources without stopping it.
- 27. 6. MONITORING
- 28. 6. MONITORING - SPARK Spark ui Spark ui rest API : http://localhost:4040/api/v1 JMX with DropWizard
- 29. 6. MONITORING - KAFKA Kafka connect REST API : to monitor Tasks Jmx metrics Connector Metrics Task Metrics Worker Metrics Worker Rebalance Metrics See KIP-196
- 30. 10 POINTS TO NOT FORGET (FROM OUR EXPERIENCE) 1. It depends on what kind of datasource & datasink you’re using. 2. Always check the Receiver (Spark Streaming) & The connector (Kafka Connect) 3. The monitoring is pretty good for both 4. Integration Test is challenging with Kafka Connect 5. You want to add more tasks, go for Kafka Connect 6. You want dynamic allocation, go for Spark Streaming 7. Deploying Spark Streaming app is simple 8. Naming conventions in Kafka Connect Framework may be confusing with the naming in Kafka core 9. You prefer conf over code, go for Kafka Connect 10. You want a distributed, copy from or to Kafka, without using a cluster Manager : Go for Kafka Connect AS ALWAYS IT REALLY DEPENDS ON YOUR NEEDS ☺
- 31. THANK YOU