Ingest and Stream Processing - What will you choose?
Download as PPTX, PDF2 likes3,739 views
The document discusses various streaming ingestion and processing engines, including Kafka, Flume, Storm, Spark Streaming, and Flink, highlighting their features, strengths, and use cases. It emphasizes concepts like low latency, high throughput, and unique handling of data, while also mentioning tools like StreamSets for data collection. The presentation explores the differences in performance and capability among these technologies, advocating for consideration of specific requirements when choosing a streaming solution.
Ingest and Stream Processing - What will you choose?
- 1. 1© Cloudera, Inc. All rights reserved. 13 June2016 Ted Malaska| Principle Solutions Architect @ Cloudera, Pat Patterson| Community Champion @ StreamSets Ingest and Stream Processing - What will you choose?
- 2. 2© Cloudera, Inc. All rights reserved. About Ted and Pat Ted Malaska • Principal Solutions Architect @ Cloudera • Apache HBase SparkOnHBase Contributor • Contact • ted.malaska@cloudera.com • @TedMalaska Pat Patterson • Community Champion @ StreamSets • Formerly Developer Evangelist at Salesforce • Contact • pat@streamsets.com • @metadaddy
- 3. 3© Cloudera, Inc. All rights reserved. Streaming Patterns •Ingestion •Low Millisecond Actions •Near Real Time Complex Actions
- 4. 4© Cloudera, Inc. All rights reserved. Parts Of Streaming Producer Kafka Engine Destination
- 5. 5© Cloudera, Inc. All rights reserved. Parts Of Streaming Producer Kafka Engine Destination At Least once Ordered Partitioned At Least Once Depends Depends
- 6. 6© Cloudera, Inc. All rights reserved. Destinations • File Systems: example HDFS • Batch is good • Only can do exactly once is a file is closed in a single ack. • Good for Scans • Solr • Everything is Document based making exactly once • Batch is still good • Good for Search Queries
- 7. 7© Cloudera, Inc. All rights reserved. Destinations • NoSQL: example HBase • Everything has a row key making exactly once for writes • Increments can be applied twice is so be careful • Good for gets and puts • Kudu • Everything has a row key making exactly once for writes • Good for gets, puts, and scans
- 8. 8© Cloudera, Inc. All rights reserved. Ingestion Destinations • File Systems: example HDFS • Flume • Kafka Connect • Solr • Flume • Any Streaming Engine
- 9. 9© Cloudera, Inc. All rights reserved. Ingestion Destinations • NoSQL: example HBase • Flume • Any Streaming Engine: Storm and Spark Streaming Tested • Kudu • Flume • Kafka Connect • Any Streaming Engine: Spark Streaming Tested
- 10. 10© Cloudera, Inc. All rights reserved. Tricks With Producers • Send Source ID (requires Partitioning In Kafka) • Seq • UUID • UUID plus time • Partition on SourceID • Watch out for repartitions and partition fail overs
- 11. 11© Cloudera, Inc. All rights reserved. Streaming Engines • Consumer • Flume, KafkaConnect, Streaming Engine • Storm • Spark Streaming • Flink • Kafka Streams
- 12. 12© Cloudera, Inc. All rights reserved. Consumer: Flume, KafkaConnect • Simple and Works • Low latency • High throughput • Interceptors • Transformations • Alerting • Ingestions
- 13. 13© Cloudera, Inc. All rights reserved. Consumer: Streaming Engines • Not so great at HDFS Ingestion • But great for record storage systems • HBase • Cassandra • Kudu • SolR • Elastic Search
- 14. 14© Cloudera, Inc. All rights reserved. Storm • Old Gen • Low latency • Low throughput • At least once • Around for ever • Topology Based
- 15. 15© Cloudera, Inc. All rights reserved. Spark Streaming • The Juggernaut • Higher Latency • High Through Put • Exactly Once • SQL • MlLib • Highly used • Easy to Debug/Unit Test • Easy to transition from Batch • Flow Language • 600 commits in a month and about 100 meetups
- 16. 16© Cloudera, Inc. All rights reserved. Spark Streaming DStream DStream DStream Single Pass Source Receiver RDD Source Receiver RDD RDD Filter Count Print Source Receiver RDD RDD RDD Single Pass Filter Count Print First Batch Second Batch
- 17. 17© Cloudera, Inc. All rights reserved. DStream DStream DStream Single Pass Source Receiver RDD Source Receiver RDD RDD Filter Count Print Source Receiver RDD partitions RDD Parition RDD Single Pass Filter Count Pre-first Batch First Batch Second Batch Stateful RDD 1 Print Stateful RDD 2 Stateful RDD 1 Spark Streaming
- 18. 18© Cloudera, Inc. All rights reserved. Flink • I’m Better Than Spark Why Doesn’t Anyone use me • Very much like Spark but not as feature rich • Lower Latency • Micro Batch -> ABS • Asynchronous Barrier Snapshotting • Flow Language • ~1/6th the comments and meetups • But Slim loves it
- 19. 19© Cloudera, Inc. All rights reserved. Flink - ABS Operator Buffer
- 20. 20© Cloudera, Inc. All rights reserved. Operator Buffer Operator Buffer Flink - ABS Barrier 1A Hit Barrier 1B Still Behind
- 21. 21© Cloudera, Inc. All rights reserved. Operator Buffer Flink - ABS Both Barriers Hit Operator Buffer Barrier 1A Hit Barrier 1B Still Behind
- 22. 22© Cloudera, Inc. All rights reserved. Operator Buffer Flink - ABS Both Barriers Hit Operator Buffer Barrier is combined and can move on Buffer can be flushed out
- 23. 23© Cloudera, Inc. All rights reserved. Kafka Streams • The new Kid on the Block • When you only have Kafka • Low Latency • High Throughput • Not exactly once • Very Young • Flow Language • Very different hardware profile then others • Not widely supported • Not widely used • Worries about separation of concern
- 24. 24© Cloudera, Inc. All rights reserved. Summary about Engines • Ingestion • Flume and KafkaConnect • Super Real Time and Special • Consumer • Counting, MlLib, SQL • Spark • Maybe future and cool • Flink and KafkaStreams • Odd man out • Storm
- 25. 25© Cloudera, Inc. All rights reserved. Abstractions Code Abstractions Beam SQL Abstraction SQL UI Abstraction StreamSets Streaming Engines
- 26. 26© Cloudera, Inc. All rights reserved. StreamSets Data Collector Building a Higher Level, Open Source Tool
- 27. 27© Cloudera, Inc. All rights reserved. Traditional and Big Data Founders StreamSets Company Background Top tier Investors Momentum to Date Strategic Partners • Founded 2014; exited stealth 9/15 • ~30 employees • Double-digit enterprise customers • 10,000 downloads
- 28. 28© Cloudera, Inc. All rights reserved. Thank you!
Editor's Notes
- #28: StreamSets was founded in 2014 by Informatica and Cloudera veterans. Learnings from big data and data integration. Mission to accelerate time to analysis by bringing deep introspection to data in motion. Launched company and initial product in September 2015. Especially well received by Cloudera and Elastic ecosystems