Data analytics at scale implementing stateful stream processing - publish

Data Analytics @ Scale:
Implementing stateful stream processing
Michael Kanevsky
VP Innovation
michaelk@codevalue.net
@mkanevsky
http://codevalue.net

About me
Michael Kanevsky
 Distributed Systems
 Big Data & Machine Learning
 Embedded & IoT
2
𝑓 =“cat”

Agenda
 Why now?
 Streaming basics
 Stream processing frameworks 101
 Building a streaming application
 A look at Apache Flink + Kafka Streams
3

Big & Fast Data
 Typical “Big Data” challenges
 Data Volume
 High velocity data streams
 Result relevance
 Offline vs Online
4
1TB
1TB

Scale: Data vs Latency
5
TBs
GBs
MBs
1s 1ms10ms100ms 100us10s
Piece
of
Cake!
Mostly
Feasible
Good
Luck…
 Improving throughput = scaling out
 Communication latencies
 Much harder for some applications
Datasize
Processing
latency

Many natural streaming applications
IOT: Telemetry & sensor data
User activities & clickstreams
Financial data processing
Calculating aggregates on data
6

What makes a good case for streaming?
Continuous data
Time anchoring
Low latency requirement
7

Architecting for analytic workload
8
Processing 1: Store
File 1 File 2 File N
…
Processing 2: Batch
KV Store
6
5
{
“customerId”: 12345,
“transactionSum”: 99.90,
“vendorId”: ”04ec092a-421a-4481”,
“ts”: ”2019-05-30T09:20:50.100+02:00”
}

Architecting for analytic workload #2
9
Processing 1: Store
File 1 File 2 File N Processing 2: Batch
KV Store
6
5
Stream Processing
KV Store
4
4
3
Partitioner
SUM
AVG
Last 30 min

Lambda Architecture
10
System complexity
Duplicate data state
Cascading failures
But get:
Low latency
Accuracy
Complex persistent state

Stream Processing Infrastructures
11
Apache
Gearpump

Our Scenario: Fraud detection
12
Event Sources Alerts
Persistence
Business Logic
Client API
- CustomerId (guid)
- Timestamp
- Transaction Sum
- Vendor Id
Notifications

Spark Streaming
14
 Supports two APIs:
 Spark Streaming (RDD-like Dstream)
 Structured Streaming (DataFrames/Datasets)
 Better in cloud
 Easier code reuse between batch and stream
 Supports Java/Scala/Python/R/SQL
 Requires dedicated cluster (RM/containerized SA)
 JAR-based execution
Zookeeper cluster
Kafka Broker
Resource Manager
Spark Cluster
Shared
Persistence
Master WorkerWorker

Apache Flink
- In Apache incubation since 2014
- Versatile API stack
- Exactly once message processing
- High throughput + low latency
- Flexible state persistence (RocksDB)
- Stream & Batch Semantics
16
JobManager
JobManager
TaskManager
TaskManager
Slot I
Slot II
….
Zookeeper cluster
Storage

Apache Flink processing modes
17
Stateful Stream Processing
DataStream API
SQL
Table API
Application
…
stream
.keyBy(“customerId”)
.timeWindow(Time.seconds(30))
.sum(“transactionSum”)
Table customerTransactions = …
Table transactionAggregates = customerTransactions
.select(“customerId, transactionSum,ts”)
.window(
Tumble.over(“30.seconds”)
.on(“ts”)
.as(“hmWindow”))
.groupBy(“hmWindows, customerId”)
.select(“transactionSum.sum as totalAmount”)
tableEnv.registerDataStream(“transactions",
“customerId, transactionSum, ts");
Table result = tableEnv.sqlQuery(
"SELECT customerId, TUMBLE_END(ts, INTERVAL ‘30’
SECOND), SUM(transactionSum) AS totalAmount
FROM transactions
GROUP BY (customerId, TUMBLE(ts, INTERVAL ‘30'
SECOND)) ";

Kafka Streams
18
 Built on Apache Kafka as a client library
 Can support low latency processing
 Kafka persistence + local RocksDB persistence
 Good DSL with flexible topology support
 Supports “Exactly Once” semantics
 Limited to Kafka -> Kafka use cases
 Available from major 0.10, fully featured as of Kafka 2.X
Zookeeper cluster
Kafka Broker
Kafka Streams App
Local
Persistence

Time -> Data Windows
19
Tumbling Windows Sliding Windows

Stream Processing: Time dimension
Two approaches:
 True Streaming
 Micro batching
Event time
Processingtime
Ideal
In practice
Three timelines:
 Event time
 Ingest time
 Processing time
One watermark!
(to rule them all)

Stream Processing: High Availability
23
Stream Processing
JobManager
Task Executor
Task Executor
Task Executor
Task Executor
Task Executor
Task Executor
State Persistence
APIMSG MSGMSG
MSG
MSG
MSG
Job
Persistence

Stream Processing: Delivery Guarantees
24
System guarantees:
 At Most Once
 At Least Once
 Exactly Once
Message
Processing
Failure
Give upTry again

Streaming Architecture
25
Distributed Queue
Kafka/Kinesis/Event Hub/…
Event
Publishers
μServices Stream Processing
File/BLOB
KV Store
MQ
Data Source
Data Source
Processor
Processor
Data Sink
Data Sink
State
Persistence
Data
Persistence

Stream Processing Frameworks
27
Streaming API Infra State Batch support
μ-batches
Framework
SA + RM In memory (by default) via Spark
kafka streams
Native
Library Kafka
+
Persisted (RocksDB)
n/a
Apache Flink Native
Framework
SA + RM Persisted (RocksDB) Out of the box
.*

Advanced features: State API
28
• Flink provides access to state via Queryable state client [Beta]
• Kafka Stream supports interactive queries to state, including local
window stores

Advanced features: Checkpoints & Snapshots
29
Checkpointing = Distributed state persistence
- Main use is used for resilience and resumable processing
- Specifies a point in data flow that synchronizes all states
In Spark state is persisted internally (to HashMap backed to persisted storage)
Kafka Streams uses Kafka for persistence + RocksDB for local state
Flink uses RocksDB for state + provides snapshots API
Data Flow inside Flink with checkpoint indicatorsData Flow

- Ideas behind Streaming
- Architecture of stream processing
- Streaming frameworks overview
- Flink example
- Kafka Streams example
30
Recap

Michael Kanevsky
VP Innovation
michaelk@codevalue.net
http://codevalue.net

Data analytics at scale implementing stateful stream processing - publish

More Related Content

What's hot (20)

Similar to Data analytics at scale implementing stateful stream processing - publish (20)

More from CodeValue (20)

Recently uploaded (20)

Data analytics at scale implementing stateful stream processing - publish

Editor's Notes