Windowing in Kafka Streams and Flink SQL

Windowing in Kafka Streams and Flink SQL
Bill Bejeck, Staff DevX Engineer
Apache Kafka committer and PMC member
bill@confluent.io | @bbejeck

Wanna Buy a Book?
“Kafka Streams in Action” - 2nd edition published!
2

@bbejeck
Kafka Streams – DSL API
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String > stockStr = builder.stream(”stocks”);
stockStr.groupByKey()
.windowedBy(...)
.aggregate(()-> new TradeStats(),
(k, v, tradeStats) -> tradeStats.add(v))
.toStream()
.map(..)
.to(“output”);
4

@bbejeck
Processor Topology
builder.stream(..)
.windowedBy()
.aggregate()
map()
to(..)
5

Architecture Overview - Flink
8
@bbejeck

9
@bbejeck
Task Distribution & Assignment – Flink

Architecture Overview – Flink SQL
10
@bbejeck

11
@bbejeck
Architecture Overview – Flink SQL

13
@bbejeck
Otter and Squirrel Wildlife Refuge

@bbejeck
Use Case
“Every 30 seconds give me the average temperature of the
squirrel and otter pens over the last minute”
19

@bbejeck
Kafka Streams – Hopping Window
.windowedBy(
TimeWindows
.ofSizeWithNoGrace(Duration.ofMinutes(1)
.advanceBy(Duration.ofSeconds(30))
)
.aggregate(..)
...
22

@bbejeck
Flink SQL – Windowing TVF
23

@bbejeck
Flink SQL – Hopping Window
SELECT window_start, window_end, device_id,AVG(reading) AS avg_reading
FROM TABLE(HOP (
TABLE device_readings,
DESCRIPTOR(ts),
INTERVAL '30' SECONDS,
INTERVAL '1' MINUTES
))
GROUP BY window_start, window_end, device_id
24

@bbejeck
Use Case
“Every hour give me a count of visitors entering the
park”
26

@bbejeck
Kafka Streams – Tumbling Window
.windowedBy(
TimeWindows
.ofSizeWithNoGrace(Duration.ofMinutes(1))
)
.aggregate(..)
...
29

@bbejeck
Flink SQL – Tumbling Window
SELECT window_start, window_end, device_id,AVG(reading) AS avg_reading
FROM TABLE(TUMBLE (
DESCRIPTOR(ts),
))
GROUP BY window_start, window_end, device_id
30

@bbejeck
Use Case
“Give me total food consumption every hour, updated at
15-minute intervals”
32

@bbejeck
Cumulating Windows
33

@bbejeck
Cumulating Windows
34

@bbejeck
Cumulating Windows
SELECT window_start, window_end, user_id, SUM(page_view)
FROM TABLE(CUMULATE (
DESCRIPTOR(ts),
INTERVAL ‘15' SECONDS,
))
GROUP BY window_start, window_end, user_id
35

@bbejeck
Use Case
“Give me a rolling average of power usage from devices
that report within 30 seconds of each other”
37

@bbejeck
Kafka Streams – Sliding Window
.windowedBy(
SlidingWindows.ofTimeDifferenceWithNoGrace
(Duration.ofMinutes(1))
)
.aggregate(..)
...
39

@bbejeck
Flink SQL – OVER Aggregation
40

@bbejeck
41

@bbejeck
42

@bbejeck
SELECT device_id, report_time,
AVG(temp_reading) OVER (
PARTITION BY location
ORDER BY report_time
RANGE BETWEEN INTERVAL '1' MINUTE PRECEDING
AND CURRENT ROW
) AS one_minute_location_temp_averages
FROM readings;
43

@bbejeck
Use Case
“I need to track the otter usage of the pool”
“How do customers interact with our website?”
45

@bbejeck
Kafka Streams - Session Window
.windowedBy(
SessionWindows.ofInactivityGapWithNoGrace(
Duration.ofMinutes(1))
)
.aggregate(..)
...
47

@bbejeck
Flink SQL – Session Window
SELECT window_start, window_end COUNT(click) AS total_clicks
FROM TABLE(SESSION (
DESCRIPTOR(ts),
))
GROUP BY window_start, window_end;
48

50
@bbejeck
Time Semantics – Stream Processing

51
@bbejeck
Time Semantics – Alignment

52
@bbejeck

53
@bbejeck

54
@bbejeck
Time Semantics – Advancement

55
@bbejeck
Time Semantics – Advancement Kafka Streams

56
@bbejeck
Time Semantics – Advancement Flink SQL

@bbejeck
Flink SQL – Specify Watermarks
CREATE TABLE ratings (
rating_id INT,
title STRING,
release_year INT,
rating DOUBLE,
rating_time TIMESTAMP(3),
WATERMARK FOR rating_time AS rating_time
)
57

@bbejeck
Kafka Streams – Handling out of order
.windowedBy(
TimeWindows
.ofSizeAndGrace(Duration.ofMinutes(1),
Duration.ofSeconds(30))
)
.aggregate(..)
...
58

@bbejeck
Flink SQL – Handling out of order
CREATE TABLE ratings (
rating_id INT,
title STRING,
release_year INT,
rating DOUBLE,
rating_time TIMESTAMP(3),
WATERMARK FOR rating_time AS rating_time - INTERVAL '30' SECOND
)
59

61
@bbejeck
Analysis – Kafka Streams

@bbejeck
public class WindowTimeToAggregateMapper implements KeyValueMapper<..>
@Override
public KeyValue<String, Agg> apply(Windowed<String> windowed,
Agg myAgg) {
long start = windowed.window().start();
long end = windowed.window().end();
myAgg.setWindowStart(start);
myAgg.setWindowEnd(end);
return KeyValue.pair(windowed.key(), myAgg);
}
}
62

@bbejeck
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String > stockStr = builder.stream(”stocks”);
.windowedBy(...)
.aggregate(()-> new TradeStats(),
(k, v, tradeStats) -> tradeStats.add(v))
.toStream()
.map(new WindowTimeToAggregateMapper())
.to(“output”);
63

@bbejeck
Analysis - Flink SQL
SELECT location, device_id, report_time, avg_temps
FROM (
SELECT location, device_id, report_time, Avg(reading)
OVER (ORDER BY report_time
RANGE BETWEEN INTERVAL '15' MINUTE PRECEDING AND CURRENT ROW
) AS avg_temps
FROM readings
)
WHERE avg_temps > N;
64

@bbejeck
CREATE TABLE reading_alerts (location STRING,
device_id STRING,
report_time TIMESTAMP(3),
reading_alerts DOUBLE);
65

@bbejeck
INSERT INTO reading_alerts
SELECT location, device_id, report_time, avg_temps
FROM (
SELECT location, device_id, report_time, Avg(reading)
OVER (ORDER BY report_time
RANGE BETWEEN INTERVAL '15' MINUTE PRECEDING AND CURRENT ROW
) AS avg_temps
FROM readings
)
WHERE avg_temps > N;
66

@bbejeck
Testing - Kafka Streams
try (TopologyTestDriver driver = new TopologyTestDriver(topology)) {
TestInputTopic<String, String> inputTopic = driver.createInputTopic(....)
TestOutputTopic<String, Agg> outputTopic = driver.createOutputTopic(…)
Instant instant = Instant.now();
int advanceOne = 20;
int advanceTwo = 40;
}
68

@bbejeck
LocalDateTime localDateTime = LocalDateTime.of(localDate.getYear(),
localDate.getMonthValue(),
localDate.getDayOfMonth(),
12, 0, 18);
Instant instant = localDateTime.toInstant(ZoneOffset.UTC);
69

@bbejeck
inputTopic.pipeInput(key, value, instant);
inputTopic.pipeInput(key, value, instant.plusSeconds(advanceOne));
inputTopic.pipeInput(key, value, instant.plusSeconds(advanceTwo));
}
70

@bbejeck
Testing – Flink SQL
StreamTableEnvironment.executeSql(...)
TableResult tableResult = streamTableEnv.exeucuteSql(...)
71

@bbejeck
Testing – Flink SQL
List<Row> rowResults = extract(tableResult.collect())
...
List<Row> expected = [ Row.ofKind(...) ]
...
assert(rowResults.contain(expected))
72

@bbejeck
Resources
• Apache Flink® on Confluent Cloud - https://www.confluent.io/product/flink/
• Kafka Streams 101 - https://developer.confluent.io/learn-kafka/kafka-
streams/get-started/
• Apache Flink 101 - https://developer.confluent.io/courses/apache-flink/intro/
• Kafka Streams in Action - 2nd edition published!
• https://www.manning.com/books/kafka-streams-in-action-second-edition
73

Windowing in Kafka Streams and Flink SQL

More Related Content

Similar to Windowing in Kafka Streams and Flink SQL (20)

More from HostedbyConfluent (20)

Recently uploaded (20)

Windowing in Kafka Streams and Flink SQL