Time and ordering in streaming distributed systems
Download as PPTX, PDF0 likes823 views
The document discusses time and ordering in distributed streaming systems through three lenses: uniformity of time, arrow of time, and perception of time. It explains that in distributed systems, clocks are not perfectly synchronized and time is not uniform. It also discusses how the direction of time can be blurred by out of order event processing. Finally, it explores how perceptions of time and ordering depend on an observer's frame of reference, and there is no single global ordering of events in a distributed system.
Time and ordering in streaming distributed systems
- 1. Time and Ordering in Streaming Distributed Systems Zhenzhong Xu Real-time Data Infrastructure Netflix @ZhenzhongXu
- 2. Time and Ordering in Streaming Distributed Systems Zhenzhong Xu Real-time Data Infrastructure . @ZhenzhongXu
- 4. Software engineers think of time as - ● Uniformly measurable ● One directional ● Infinite precision ● Time manifests ordering of events
- 5. “Time no longer appears to us as a gigantic, world-dominating chronos, nor as a primitive entity, but as something derived from phenomena themselves. It is a figment of my thinking.” — Schrödinger, Erwin. “Time is an illusion.” — Einstein, Albert.
- 6. Distributed System No shared memory, only message passing via unreliable network with variable delays, and the system may suffer from partial failures, unreliable clocks and processing pauses.
- 7. Stream processing connects distributed systems together, over space and time, designed with unbounded data set in mind.
- 8. Stream Processing at Netflix ● Keystone Data Pipeline ● Operation insights ● Business analytics ● Event sourcing pattern
- 9. Categories of streaming ● Time agnostic ● Transformation ● Filtering ● Projection ● Enrichment ● Inner joins
- 10. Categories of streaming ● Time agnostic ● Approximation ● Approximate top-n ● Streaming k-means ● etc
- 11. Categories of streaming ● Time agnostic ● Approximation ● Windowing ● Fixed / Tumbling ● Sliding / Hopping ● Session / Dynamic
- 12. Project Delta Eventual consistent, event- driven data synchronization platform ● Event sourcing ● Windowing Challenges: ● Semantics of ordering ● Latency vs. durability ● CDC ● etc
- 13. … via the three lens of time ● Uniformity of time ● Arrow of time ● Perception of time
- 15. Time is a tool ...
- 16. Time is a tool ...
- 17. Time is a tool ...
- 18. Need for synchronization? = uniform time
- 19. Standing on the shoulder of giants
- 20. Time flows slower closer to a black hole Scene from the movie Interstella, depicts time flows slow closer to the supermassive blackhole “Gargantuan”
- 21. Clock synchronization over network “NTP can usually maintain time to within tens of milliseconds over the public Internet, and can achieve better than one millisecond accuracy in local area networks under ideal conditions. Asymmetric routes and network congestion can cause errors of 100 ms or more.” https://en.wikipedia.org/wiki/Network_Time_Protocol
- 22. Figure referenced from Designing Data Intensive Applications by Martin Kleppmann, Chapter 8 Trouble with Distributed Systems Relying on synchronized wall clock timestamps?
- 24. Why time skews ● Information travel takes time ● Low power device ● Process failure ● Unpredictable network congestions ● Timeouts and unbounded delays ● Unreliable clock ● Process pauses ● etc Figure referenced from Streaming Systems by Tyler Akidau el al, Chapter 1 Streaming 101
- 25. Watermark in action Animation referenced from Streaming Systems by Tyler Akidau el al, http://streamingbook.net/fig/2-11
- 26. Use watermark to bound the uncertainties of time ● Allowed lateness Figure referenced from Streaming Systems by Tyler Akidau el al, Chapter 2 Going Streaming: When and How
- 27. Project Delta
- 28. Watermark and allowed lateness POST arrives outside allowed lateness boundary PRE event duplicated
- 29. #2 Arrow of time
- 33. PRE ts: GMT: Saturday, November 17, 2018 7:10:00 AM POST ts: GMT: Saturday, November 17, 2018 7:09:00 AM
- 34. Stream Processing Custom windowing Embed our blurred vision to represent arrow of time in custom triggers logic Out of order event timestamp
- 36. This is a story about me and my uncle ...
- 37. Me, 4 years old My uncle, 2 years old
- 38. Imagine an ancestry tree includes all modern human beings ...
- 39. When forcing a global generation order... Can Bob and Dave be logically the same generation?
- 40. What’s the meaning of “now”?
- 41. Light travels in a cone shape over time ...
- 42. The light cone representing the past, present, and future ... https://en.wikipedia.org/wiki/Light_cone
- 45. The cone shape shows the causal/partial ordering from Dave’s frame of reference.
- 46. Lorentz transformation Observers in different frame of references perceive different ordering of events
- 47. Relativity of Simultaneity Time and Ordering depends on frame of reference (space and time!) There is no deterministic global ordering.
- 49. In a distributed system, it is sometimes impossible to say that one of two events occurred first. The relation “happened before” is therefore only a partial ordering of the events in the system.
- 50. Figure referenced from wikipedia: https://en.wikipedia.org/wiki/Vector_clock
- 51. Partial/Causal ordering An irreflexive partial ordering on a set A is a relation on A that satisfies three properties. 1. irreflexivity: a ⊀ a 2. antisymmetry: if a < b then b ⊀ a 3. transitivity: if a < b and b < c then a < c
- 52. Total ordering An irreflexive total ordering is a irreflexive partial ordering that satisfies another condition. 4. totality: if a ≠ b then a < b or b< a.
- 55. Distributed Consensus and Atomic Broadcast is the same thing! Both requires total order broadcast.
- 56. Linearizability … to make a system appear as if there is only a single copy of the data. Linearizability is the C in CAP theorem. (practically no CA system, only CP) Linearizability requires total ordering…
- 57. Consensus in a synchronous environment can be resilient to faults. FLP result shows that in an async setting, where only one processor might crash, there is no distributed algorithm that solves the consensus problem.
- 58. What happens when an event get close to a black hole’s event horizon This is very similar to how process fails in distributed systems, observer will never be able to tell whether the process crashed or simply will take long time to respond
- 60. The 3 lens of time ● No uniformity of time ● Blurred direction of time ● Limited perception of time
- 63. Thank you.