Streams and Tables: Two Sides of the Same Coin (BIRTE 2018)
6 likes1,210 views
The document discusses streams and tables as two sides of the same coin. It proposes a dual streaming model that handles out-of-order data by continuously updating stateful operators and emitting changelog streams. This allows processing data with low latency while avoiding buffering and reordering. The model is implemented in Apache Kafka Streams and adopted in industry through tools like KSQL.
![25
References
[1] Arvind Arasu, Shivnath Babu, and Jennifer Widom. 2003. CQL: A Language for Continuous Queries
over Streams and Relations. Database Programming Languages, 9th Int. WS. 1–19.
[2] Badrish Chandramouli et al. 2014. Trill: A High-performance Incremental Query Processor for
Diverse Analytics. Proc. VLDB Endow. 8, 4 (2014), 401–412.
[3] Jin Li et al. 2005. Semantics and Evaluation Techniques for Window Aggregates in Data Streams.
Proc. of the ACM SIGMOD Int. Conf. on Management of Data. 311–322.
[4] Sailesh Krishnamurthy et al. 2010. Continuous Analytics over Discontinuous Streams. Proc. of the
2010 ACM SIGMOD Int. Conf. on Management of Data. 1081–1092.
[5] Yan-Nei Law, HaixunWang, and Carlo Zaniolo. 2004. Query Languages and Data Models for
Database Sequences and Data Streams. Proc. of the 13th Int. Conf. on Very Large Data Bases. 492-503.
[6] Shivnath Babu and Jennifer Widom. 2001. Continuous Queries over Data Streams. SIGMOD Records
30, 3 (2001), 109–120.
[7] Daniel Abadi et al. 2005. The Design of the Borealis Stream Processing Engine. CIDR, 2nd Biennial
Conf. on Innovative Data Systems Research. 277–289.](https://image.slidesharecdn.com/birte2018published-180829162554/85/Streams-and-Tables-Two-Sides-of-the-Same-Coin-BIRTE-2018-25-320.jpg)
Streams and Tables: Two Sides of the Same Coin (BIRTE 2018)
- 1. 1 Streams and Tables: Two Sides of the Same Coin Matthias J. Sax12, Guozhang Wang1, Matthias Weidlich2, Johann-Christoph Freytag2 1Confluent Inc., Palo Alto (CA) matthias@confluent.io guozhang@confluent.io 2Humboldt-Universität zu Berlin mjsax@informatik.hu-berlin.de matthias.weidlich@hu-berlin.de freytag@informatik.hu-berlin.de @MatthiasJSax Twelfth International Workshop on Real-Time Business Intelligence and Analytics 27 August, 2018, Rio de Janeiro
- 2. 2 Count Clicks per Page BIRTE VLDB VLDB Distributed Data Source BIRTE(3) VLDB(4) VLDB(7) BIRTE(3) VLDB(4) VLDB(7) BIRTE(3)VLDB(4)VLDB(7) Input Data Stream
- 3. 3 Count Clicks per Page BIRTE VLDB VLDB Distributed Data Source BIRTE(3) VLDB(4) VLDB(7) BIRTE(3) VLDB(4) VLDB(7) BIRTE(3) VLDB(4)VLDB(7) Input Data Stream Arrival order non-deterministic Even-time semantics implies out-of-order data
- 4. 4 Ordering: Common Approaches Input Data Stream BIRTE(3) VLDB(4)VLDB(7) BIRTE(3)VLDB(4)VLDB(7) buffer and re-order SPS SPS punctuations/watermarks BIRTE(3) VLDB(4)VLDB(7) time=3
- 5. 5 Cost Correctness/ Completeness Latency Buffering and Reordering - Ref: CQL1, Trill2 Punctuations/Watermarks - Ref: Li et al.3, Krishnamurthy et al.4 Design Space
- 6. 6 Problem Statement How to design a model • for the evaluation of expressive operators • with low latency over potentially unordered data streams • that can be implemented by mean of distributed online algorithms?
- 7. 7 High-Level Proposal • To reduce latency, we need to avoid any processing delays • Process data in arrival order • Emit current result immediately • Law et al.5: cannot handle out-of-order data • To handle out-of-order data, we need to be able to update/refine previous results • Data streams must allow for update records • Update/delete records by Babu and Widom6: no operator semantics defined • Borealis:7 replays data stream after “updating/reordering”; very high cost
- 8. 8 Data Model • Offset: physical order (arrival/processing order) • Timestamp: logical order (event-time) • Key: optional for grouping • Value: payload
- 9. 9 Stream Processing Operators • Stateless, order agnostic • filter, projection, flatMap • No special handling necessary • Stateful, order sensitive • aggregation, joins, windowing • Need to handle out-of-order data
- 10. 10 Data Stream Aggregation • Model output of (windowed) aggregations as table • State is not internal but first-class citizen • Update stateful operator continuously • Emit changelog stream to downstream operators • Streams, Table, and Changelogs • Define operator semantics over changelogs and updating tables • Temporal operator semantics
- 11. 11 Example: Count Clicks per Page url count record stream changelog stream BIRTE 1 <BIRTE,1> BIRTE 1 countTable = stream.groupBy(r->url).count()
- 12. 12 Example: Count Clicks per Page url count record stream changelog stream 1 <BIRTE,1> BIRTE 1 VLDB 1VLDB <VLDB,1> countTable = stream.groupBy(r->url).count() VLDB
- 13. 13 Example: Count Clicks per Page url count record stream changelog stream 1 <BIRTE,1> BIRTE 1 VLDB 2VLDB <VLDB,2> countTable = stream.groupBy(r->url).count() VLDB <VLDB,1>
- 14. 14 countTable2 = countTable.filter(url=‘VLDB’).toTable() Example: Processing a Changelog Stream url count changelog stream 1 <BIRTE,1> BIRTE VLDB <VLDB,1> 2 <VLDB,2> url count 2VLDB
- 15. 15 countTable = stream.groupBy(r->url).windowedBy(5sec).count() windowID = <groupingKey,windowStartTimestamp> windowStartTimestamp = recordTimestamp / windowSize Example: Windowed Count window ID count record stream changelog stream BIRTE(3) 1 <<BIRTE,0>,1> <BIRTE,0> VLDB(7) 1<VLDB,0> <<VLDB,0>,1>VLDB(4) <VLDB,5> <<VLDB,5>,1>1
- 16. 16 countTable = stream.groupBy(r->url).windowedBy(5sec).count() windowID = <groupingKey,windowStartTimestamp> windowStartTimestamp = recordTimestamp / windowSize Example: Out-of-Order Data window ID count record stream changelog stream BIRTE(1) 2<BIRTE,0> 1<VLDB,0> <VLDB,5> 1 <<BIRTE,0>,2>
- 17. 17 Duality of Streams and Tables
- 18. 18 Cost Correctness/ Completeness Latency Buffering and Reordering - Ref: CQL1, Trill2 Punctuations/Watermarks - Ref: Li et al.3, Krishnamurthy et al.4 Design Space Dual Streaming Model - continuous updates / changelogs - decouple latency from correctness - trade-off latency and cost - trade-off cost and completeness (retention time)
- 19. 19 Stream-Table Transformations See the paper for details…
- 20. 20 Implementation • Implemented in Apache Kafka (v0.10) • Kafka Streams / Streams API StreamsBuilder builder = new StreamsBuilder(); KStream<String, String> textLines = builder.stream("TextLinesTopic"); KTable<String, Long> wordCounts = textLines .flatMapValues(textLine -> Arrays.asList(textLine.toLowerCase().split("W+"))) .groupBy((key, word) -> word) .windowedBy(TimeWindows.of(5_000L)) .count(); wordCounts.toStream().to("WordsWithCountsTopic"); KafkaStreams streams = new KafkaStreams(builder.build(), props); streams.start();
- 21. 21 Implementation • Implemented in Apache Kafka (v0.10) • Kafka Streams / Streams API • Leveraged in Confluent’s KSQL CREATE TABLE click_count_per_url AS SELECT url, count(*) FROM click_stream WINDOW TUMBLING (SIZE 1 MINUTE) WHERE url LIKE = '%confluent%' OR url LIKE ‘%hu-berlin%’ GROUP BY url;
- 22. 22 Implementation • Implemented in Apache Kafka (v0.10) • Kafka Streams / Streams API • Leveraged in Confluent’s KSQL • Widely adopted in industry
- 23. 23 Summary • Suggest the Dual-Streaming-Model • Handles out-of-order data within the processing model • Optimized for low latency • Streams and Tables are Dual • Allows to trade-off processing cost, latency, completeness • Adopted in industry via Kafka Streams and KSQL
- 24. 24 Thank You We are hiring!
- 25. 25 References [1] Arvind Arasu, Shivnath Babu, and Jennifer Widom. 2003. CQL: A Language for Continuous Queries over Streams and Relations. Database Programming Languages, 9th Int. WS. 1–19. [2] Badrish Chandramouli et al. 2014. Trill: A High-performance Incremental Query Processor for Diverse Analytics. Proc. VLDB Endow. 8, 4 (2014), 401–412. [3] Jin Li et al. 2005. Semantics and Evaluation Techniques for Window Aggregates in Data Streams. Proc. of the ACM SIGMOD Int. Conf. on Management of Data. 311–322. [4] Sailesh Krishnamurthy et al. 2010. Continuous Analytics over Discontinuous Streams. Proc. of the 2010 ACM SIGMOD Int. Conf. on Management of Data. 1081–1092. [5] Yan-Nei Law, HaixunWang, and Carlo Zaniolo. 2004. Query Languages and Data Models for Database Sequences and Data Streams. Proc. of the 13th Int. Conf. on Very Large Data Bases. 492-503. [6] Shivnath Babu and Jennifer Widom. 2001. Continuous Queries over Data Streams. SIGMOD Records 30, 3 (2001), 109–120. [7] Daniel Abadi et al. 2005. The Design of the Borealis Stream Processing Engine. CIDR, 2nd Biennial Conf. on Innovative Data Systems Research. 277–289.
- 27. 27 Evolving Table window ID count 1<BIRTE,0> 1<VLDB,0> window ID count 1<BIRTE,0> window ID count 1<BIRTE,0> 1<VLDB,0> <VLDB,5> 1 record stream BIRTE(3)VLDB(7) VLDB(4)BIRTE(1) table v3 table v4 table v7
- 28. 28 Evolving Table window ID count <BIRTE,0> 1<VLDB,0> window ID count <BIRTE,0> window ID count <BIRTE,0> 1<VLDB,0> <VLDB,5> 1 record stream BIRTE(3)VLDB(7) VLDB(4)BIRTE(1) table v3 table v4 table v7 window ID count 1<BIRTE,0> table v1 2 2 2
- 30. 30 Stream-Stream Join • Sliding Window Join, i.e., band join • Window size specifies additional timestamp based join predicate SELECT * FROM stream1, stream2 WHERE stream1.key = stream2.key AND stream1.ts – windowSize <= stream2.ts AND stream2.ts <= stream1.ts + windowSize windowSize stream1 stream2
- 31. 31 Stream-Table Join • Temporal table “lookup” join • For each stream record, lookup for a matching table record • Join condition: streamRecord.key == tableRecord.key • The join is temporal is the sense, that the “correct” table version must be use • i.e., youngest table version that is before the stream records timestamp