Stream Data Processing at Big Data Landscape by Oleksandr Fedirko
0 likes329 views
This document provides an overview of stream data processing and common stream processing tools. It discusses streaming basics like stateful and stateless operations. It also covers microbatch vs realtime streaming and compositional vs declarative stream processing engines. Typical stream processing architectures and use cases are presented. Main considerations for projects using stream processing are outlined. An overview of popular stream processing tools like Apache Spark, Storm, Flink, and cloud services from AWS, GCP and Azure is provided. The document concludes with a case study example and questions for discussion.
Stream Data Processing at Big Data Landscape by Oleksandr Fedirko
- 2. 2 Confidential Stream data processing at BigData landscape
- 3. 3 Confidential Intro Meet your speaker today: Oleksandr Fedirko - CEE Head of Big Data Practice
- 4. 4 Confidential High level agenda Streaming basics Types of stream systems Typical architectures and use cases Main consideration on a project with Stream processing Stream processing tools overview Case study Q&A session
- 6. 6 Confidential Streaming basics Types of streaming operations - Stateful - Aggregation - Join - Sorting - Stateless - Filter - Map
- 7. 7 Confidential Streaming basics Types of streaming sources - Bounded - Database - Flat file - Key-value storage - Unbounded - Queue - Port - Socket
- 12. 12 Confidential 12 Types of stream systems
- 13. 13 Confidential MicroBatches vs Realtime streaming Micro Batches - Most of the tools/frameworks work under this paradigm - Widely used, mature ecosystem Realtime streaming - Better performance with stateless operations - Can fulfill particular use cases where low latency is a must
- 14. 14 Confidential Compositional vs Declarative engines In a compositional stream processing engines, developers define the Directed Acyclic Graph (DAG) in advance and then process the data. This may simplify code, but also means developers need to plan their architecture carefully to avoid inefficient processing. Challenges: Compositional stream processing are considered the “first generation” of stream processing and can be complex and difficult to manage. Examples: Compositional engines include Samza, Apex and Apache Storm.
- 15. 15 Confidential Compositional vs Declarative engines Developers use declarative engines to chain stream processing functions. The engine calculates the DAG as it ingests the data. Developers can specify the DAG explicitly in their code, and the engine optimizes it on the fly. Challenges: While declarative engines are easier to manage, and have readily-available managed service options, they still require major investments in data engineering to set up the data pipeline, from source to eventual storage and analysis. Examples: Declarative engines include Apache Spark and Flink, both of which are provided as a managed offering.
- 16. 16 Confidential 16 Typical architectures and use cases
- 17. Typical architectures and use cases Source 1 Source 2 Source 3 Ingestion Stream processing Queue Data Lake
- 18. Source 1 Source 2 Source 3 Stream processing Queue Data Lake Typical architectures and use cases
- 19. Source 1 Source 2 Source 3 Stream processing Queue Key-value/ Columnar storage Typical architectures and use cases
- 20. Source 1 Source 2 Source 3 Stream processing Queue Typical architectures and use cases
- 21. Source 1 Source 2 Source 3 Stream processing Queue DB/Cache/ API call Typical architectures and use cases
- 22. 22 Confidential 22 Main consideration on a project with Stream processing
- 23. 23 Confidential Main consideration on a project with Stream processing Think of the next NFRs: ● Records per second, avg ● Records per second, max (spike) ● Spike longevity ● 95% of the size of record ● 1% max of the size of record ● Latency ● Exactly one/at least one/at most one semantic ● Late arrivals ● Static/dynamic streams
- 24. 24 Confidential 24 Stream processing tools overview
- 25. 25 Confidential Stream processing tools overview Apache Spark Spark is an open-source distributed general-purpose cluster computing framework. Spark’s in-memory data processing engine conducts analytics, ETL, machine learning and graph processing on data in motion or at rest. It offers high-level APIs for the programming languages: Python, Java, Scala, R, and SQL. The Apache Spark Architecture is founded on Resilient Distributed Datasets (RDDs). These are distributed immutable tables of data, which are split up and allocated to workers. The worker executors implement the data. The RDD is immutable, so the worker nodes cannot make alterations; they process information and output results.
- 26. 26 Confidential Stream processing tools overview Pros: Apache Spark is a mature product with a large community, proven in production for many use cases, and readily supports SQL querying. Cons: ● Spark can be complex to set up and implement ● It is not a true streaming engine (it performs very fast batch processing) ● Limited language support ● Latency of a few seconds, which eliminates some real-time analytics use cases
- 27. 27 Confidential Stream processing tools overview Apache Storm Apache Storm has very low latency and is suitable for near real time processing workloads. It processes large quantities of data and provides results with lower latency than most other solutions. The Apache Storm Architecture is founded on spouts and bolts. Spouts are origins of information and transfer information to one or more bolts. This information is linked to other bolts, and the entire topology forms a DAG. Developers define how the spouts and bolts are connected.
- 28. 28 Confidential Stream processing tools overview
- 29. 29 Confidential Stream processing tools overview Pros: ● Probably the best technical solution for true real-time processing ● Use of micro-batches provides flexibility in adapting the tool for different use cases ● Very wide language support Cons: ● Does not guarantee ordering of messages, may compromise reliability ● Highly complex to implement
- 30. 30 Confidential Stream processing tools overview Apache Flink Flink is based on the concept of streams and transformations. Data comes into the system via a source and leaves via a sink. To produce a Flink job Apache Maven is used. Maven has a skeleton project where the packing requirements and dependencies are ready, so the developer can add custom code. Apache Flink is a stream processing framework that also handles batch tasks. Flink approaches batches as data streams with finite boundaries.
- 31. 31 Confidential Stream processing tools overview Pros: ● Stream-first approach offers low latency, high throughput ● Real entry-by-entry processing ● Does not require manual optimization and adjustment to data it processes ● Dynamically analyzes and optimizes tasks Cons: ● Some scaling limitations ● A relatively new project with less production deployments than other frameworks
- 32. 32 Confidential Stream processing tools overview (cloud) ● AWS Kinesis ● GCP DataFlow ● Azure Stream Analytics When do we use Lambda-like application instead of services above? Very light weight simple logic.
- 34. Confidential Case study (CEP for custom DSL) Raw events Parsed events Canonically parsed events Indicators Incidents Archive job Parse job Index job Archive storage Primary storage Index job Rules job Secondary storage Application storage Save incind job Message Queues Processing Engines Sink Storages
- 36. 36 Confidential FAQ I do my custom Java based application that does consume messages from Kafka. Is it stream or not ? If I have 1 message per day in my Kafka topic could it be considered as a stream ? I love my Kafka Stream API. Why didn’t you cover it ? I have a … tool on my project. Why didn’t you mention it today ? Did you cover everything Stream related today ? Am I a Stream master after this event ?