![CQL in 5 6 Slides
Stream-to-Relation Operators:
• Sliding Window:
FROM S [ RANGE 5 Minutes]
• Parametric Sliding Windows:
FROM S [ RANGE 5 Minutes Slide 1 Min]
• Partitioned Windows:
FROM S [PARTITIONED BY A1..An ROW m]
1 a Data Domain refers to all the values which a data element may contain.
X](https://image.slidesharecdn.com/timeseries-200209200350/85/A-Brief-History-of-Stream-Processing-31-320.jpg)
![Mapping to Kafka
Minimal
Concept Partitioned Unbounded Ordering Mutable Unique key
constraint
Schema
Topic Yes Yes Yes No No No (raw
bytes)
Stream Yes Yes Yes No No Yes
Table Yes Yes No Yes Yes Yes
Concept Kafka
Streams
KSQL Java Scala Python
Topic - - List/Stream List/Stream[(Array[Byte],
Array[Byte])]
[]
Stream KStream STREAM List/Stream List/Stream[(K, V)] []
Table KTable TABLE HashMap mutable.Map[K, V] {}
https://www.michael-noll.com/blog/2018/04/05/of-stream-
and-tables-in-kafka-and-stream-processing-part1/](https://image.slidesharecdn.com/timeseries-200209200350/85/A-Brief-History-of-Stream-Processing-63-320.jpg)
A Brief History of Stream Processing
- 1. A Brief History of Stream Processing TimeSeries Meetup, Tallin, Estonia, Europe, Earth, Milky Way, Universe …. 42
- 2. Who I Am Riccardo Tommasini Research Fellow @ UT Future AssistantProfessor Enthusiast!
- 3. Timeline 20152002 2005 2008 201820102006 2007 2014 Precision not Recall
- 4. The Origin
- 5. Timeline 20152002 2005 2008 201820102006 2007 2014 Precision not Recall
- 6. The Vision
- 7. Timeline 20152002 2005 2008 201820102006 2007 2014 Precision not Recall
- 8. Timeline 20152002 2005 2008 201820102006 2007 2014 Precision not Recall
- 10. Timeline 20152002 2005 2008 201820102006 2007 2014 Precision not Recall
- 11. Precision not Recall Timeline 20152002 2005 2008 201820102006 2007 2014 Stream Processing + AI (Deductive)
- 12. The Title
- 13. Timeline 20152002 2005 2008 201820102006 2007 2014 Precision not Recall
- 14. Timeline 20152002 2005 2008 201820102006 2007 2014 Stream Processing + AI (Inductive) Precision not Recall
- 15. Timeline 20152002 2005 2008 201820102006 2007 2014 Precision not Recall
- 17. Timeline 20152002 2005 2008 201820102006 2007 2014 Big Stream Processing Starts Precision not Recall
- 18. Timeline 20152002 2005 2008 201820102006 2007 2014 S. Murthy et al. Pulsar – Real-Time Analytics at Scale. Technical report, eBay, 2015. Summingbird: A Framework for Integrating Batch and Online MapReduce Computations. Trill: A High-Performance Incremental Query Processor for Diverse Analytics. TelegraphCQ: Continuous Dataflow Processing. NiagaraCQ: A Scalable Continuous Query System for Internet Databases. Aurora: A New Model and Architecture for Data Stream Management
- 19. Our Focus
- 21. A (Query) Language Perspective It addresses the problem of manipulating and managing data-streams. It stresses on the operations that are necessary to build applications. Some Assumptions are made on the underlying system(s). It address the problem of dealing with unbounded data. Discuss the systems’ primitives that are necessary to guarantee low-latency and fault- tolerance in presence of *uncertainty*, e.g., late arrivals. A System Perspective A new hybrid hope perspective It addresses the problems in between. How do we map the abstraction of an high-level language to the underlying system?
- 22. Goals • Exploit lesson learnt in RDBMS theory. • simple, clear, yet powerful language. • efficiently combine streams and relations. • low-latency is paramount. • take data distribution and ordering into account. • avoid implicit operator semantics. CQL DSM DFM • Correctness first with sessions support. • Latency requirements vs resource cost should dictate system choice. • Single processing model for bounded and unbounded data.
- 23. Time • DSMS Clock Time, i.e., the timestamp assigned by the DSMS. • Source Time, i.e., the timestamp assigned by the source. • Source Heartbeat are used to declare no element with lower timestamp will be sent. • Logical Order induced by the timestamp assigned by the data source. • Physical Order induced by the timestamp assigned by the system at ingestion. CQL DSM DFM • Event Time is the time at which the event itself actually occurred. • Processing Time is the time at which an event is observed during processing. • Watermark, i.e., global progress metrics that tracks the skewness between ET and PT.
- 24. Watermark
- 25. Abstractions • Streams • (Time-Varying) Relations* • Streams • Change-log Streams • Records Streams • Tables CQL DSM DFM • PCollections (bounded and unbounded)
- 26. CQL in 3 Slides A Stream S is a possibly infinite multi-set of elements <s,t> where s is a tuple belonging to the schema of S and t is a timestamp. Relation R is a set of tuples (d1, d2, ..., dn), where each element dj is a member of Dj, a data domain1. 1 a Data Domain refers to all the values which a data element may contain.
- 27. CQL in 3 4 Slides A Stream S is a possibly infinite multi-set of elements <s,t> where s is a tuple belonging to the schema of S and t is a timestamp. Relation R is a set of tuples (d1, d2, ..., dn), where each element dj is a member of Dj, a data domain1. 1 a Data Domain refers to all the values which a data element may contain. X
- 28. Ok, CQL in 4 5 Slides A Stream S is a possibly infinite multi-set of elements <s,t> where s is a tuple belonging to the schema of S and t is a timestamp. Relation R is a mapping from each time instant in T to a finite but unbounded bag of tuples belonging to the schema of R. 1 a Data Domain refers to all the values which a data element may contain. X
- 29. CQL in 5 Slides A Stream S is a possibly infinite multi-set of elements <s,t> where s is a tuple belonging to the schema of S and t is a timestamp. Relation R is a mapping from each time instant in T to a finite but unbounded bag of tuples belonging to the schema of R. 1 a Data Domain refers to all the values which a data element may contain.
- 30. CQL in 5 Slides Streams Relations … <s,τ> … <s1> <s2> <s3> infinite unbounded sequence finite bag Mapping: T ! R stream-to-relation relation-to-stream relation-to-relation Stream Relation R(t) Relational Algebra (Almost) *Stream operators Sliding windows
- 31. CQL in 5 6 Slides Stream-to-Relation Operators: • Sliding Window: FROM S [ RANGE 5 Minutes] • Parametric Sliding Windows: FROM S [ RANGE 5 Minutes Slide 1 Min] • Partitioned Windows: FROM S [PARTITIONED BY A1..An ROW m] 1 a Data Domain refers to all the values which a data element may contain. X
- 32. CQL in 6 Slides R2R operator s3 s4 s5 s6 s7 s8 s9 s10 s11 s12S s1 s2 W(ω,β) β ω t widthslide
- 33. CQL in 6 Slides Relation-to-Stream Operators: • Rstream: streams out all data in the last step • Istream: streams out data in the last step that wasn’t on the previous step, i.e. streams out what is new • Dstream: streams out data in the previous step that isn’t in the last step, i.e. streams out what is old 1 a Data Domain refers to all the values which a data element may contain.
- 34. • What results are being computed • Where in event time are being computed • When in processing time are being materialised • How “earlier results” relate to “later refinements” Reasoning about time DFM in X<42* Slides *overestimation
- 35. A Stream is represented as a possibly unbounded collection of key-value pairs, i.e., a PCollection<K,V>. PTransforms are PCollections-to-PCollections operations. DFM in X Slides (WHAT) Processing Model
- 36. DFM in X Slides (WHAT) Processing Model
- 37. DFM in X Slides (WHERE) Windowing Model
- 38. DFM in X Slides (WHERE) Windowing Model
- 39. DFM in X Slides • The DFM windowing model requires extended primitives than CQL’s one. • Window Assignment, i.e., each element is assigned to a corresponding window. • Window Merge: • Drop Timestamp: only the window interval is relevant here on • GroupBy Key: to enable parallel execution • Window Merge (the merge logic depends on the window strategy) • GroupByWindow: to ensure elements are processed in sequence window- wise. • ExpandToElements: assigned a valid timestamp to the elements. (WHERE) Windowing Model Wall of Text Disclaimer
- 40. DFM in X Slides • In order to build unaligned (apply across subset of the data) event-time windows DFM is forced to decouple the reporting of the window content. • To do so, DFM introduces an orthogonal model that allows to signal when a window is ready to be processed. (WHEN) Triggering Model
- 41. DFM in X Slides • In order to build unaligned (apply across subset of the data) event-time windows DFM is forced to decouple the reporting of the window content. • To do so, DFM introduces an orthogonal model that allows to signal when a window is ready to be processed. • Triggers solve this issue allowing DFM users to write an arbitrary logic to signal the completion of a window. (WHEN) Triggering Model
- 42. DFM in X Slides In addition to the triggering semantics, DFM introduces different refinements models to deal with late arrivals • Discarding, i.e., upon triggering, window contents are discarded and later results bear no relation to previous results. • Accumulating, i.e., upon triggering, window contents are left intact in a persistent state and later results will become a refinement to previous results • Accumulating & Retracting, i.e, it extends the accumulating semantics with retraction of the previous value. (HOW) Triggering Model (part 2) Wall of Text Disclaimer
- 43. – Do the math “X ~ 9 (7 + 2 WoT)”
- 45. ZZZZ
- 47. Dual Stream Model The design space
- 48. Should I Take a Step back?
- 49. curtesy of Emanuele Della Valle - http://emanueledellavalle.org A Conceptual View of Kafka • Producers send messages on topics • Consumers read messages from topics • Messages are key-value pairs • Topics are streams of messages • Kafka cluster manages topics
- 50. A Logical View of Kafka • Brokers are the main storage and messaging components of the Kafka cluster curtesy of Emanuele Della Valle - http://emanueledellavalle.org
- 51. Reconciling the two views of Kafka • Topics are partitioned across brokers • Producers shard messages over the partitions of a certain topic • Typically, the message key determines which Partition a message is assigned to curtesy of Emanuele Della Valle - http://emanueledellavalle.org
- 52. Topic partitioning invites distributed consumption • Different Consumers can read data from the same Topic • By default, each Consumer will receive all the messages in the Topic • Multiple Consumers can be combined into a Consumer Group • Consumer Groups provide scaling capabilities • Each Consumer is assigned a subset of Partitions for consumption curtesy of Emanuele Della Valle - http://emanueledellavalle.org
- 53. Dual Stream Model The intuition
- 54. Dual Stream Model The intuition https://www.michael-noll.com/blog/2018/04/05/of-stream- and-tables-in-kafka-and-stream-processing-part1/
- 55. Dual Stream Model The intuition https://www.michael-noll.com/blog/2018/04/05/of-stream- and-tables-in-kafka-and-stream-processing-part1/
- 56. Dual Stream Model The Truth about Streams Stream Change-log Stream Record Stream Unbounded and ordered sequence of key-value pairs A streams whose records are updates to a table A streams whose records are facts records are identified by a primary key records are not identified by a primary key
- 57. Dual Stream Model A table is a collection of table versions; one version for each point in time using the timestamp as a version number. The Truth about Tables T(1)={T5 ={⟨A,7.2⟩}} T(2) = {T5 = {⟨A,7.2⟩},T6 = {⟨B,14.7⟩}} T(3) = {T5 = {⟨A,7.2⟩}, T6 = {⟨A,8.9⟩,⟨B,14.7⟩}} T(4) = {T3 = {⟨B,12.1⟩},T5 = {⟨A,7.2⟩,⟨B,12.1⟩}, T6 = {⟨A, 8.9⟩, ⟨B, 14.7⟩}} T(5) = {T3 = {⟨B,12.1⟩},T5 = {⟨A,7.2⟩,⟨B,12.1⟩}, T6 = {⟨A, 8.9⟩, ⟨B, 14.7⟩},T8 = {⟨A, 8.9⟩, ⟨B, 16.7⟩}}
- 58. Dual Stream Model The Truth about Operations Stateless Operations
- 59. Dual Stream Model The Truth about Operations Stateful Operations
- 60. Dual Stream Model The Complete Picture https://www.michael-noll.com/blog/2018/04/05/of-stream- and-tables-in-kafka-and-stream-processing-part1/
- 61. Dual Stream Model The DSM simplifies the reasoning about the transformations, but does not solve the unboundedness problem. We still need infinite memory for processing an infinite stream. DSM introduces the retention time to make the trade-off explicit. Result Correctness vs Runtime Cost
- 63. Mapping to Kafka Minimal Concept Partitioned Unbounded Ordering Mutable Unique key constraint Schema Topic Yes Yes Yes No No No (raw bytes) Stream Yes Yes Yes No No Yes Table Yes Yes No Yes Yes Yes Concept Kafka Streams KSQL Java Scala Python Topic - - List/Stream List/Stream[(Array[Byte], Array[Byte])] [] Stream KStream STREAM List/Stream List/Stream[(K, V)] [] Table KTable TABLE HashMap mutable.Map[K, V] {} https://www.michael-noll.com/blog/2018/04/05/of-stream- and-tables-in-kafka-and-stream-processing-part1/
- 64. CQL Esper Kafka Streams Stream Duality DataFlow KSQL Flink Models & Issues SECRET Model Complex Event Processing Stream Processing + AI
- 65. Meetup, Tallin First Event April/March By Confluent, so free beers Come and talk about your Kafka experience!