Unbundling the Modern Streaming Stack With Dunith Dhanushka | Current 2022
1 like395 views
The document discusses the modern streaming stack and its evolution from classic streaming methods to current cloud-native approaches that simplify real-time analytics application development. It emphasizes the significance of event-driven architectures and the importance of rapid processing and analytics of streaming data for deriving business insights. Key takeaways include starting small, understanding technology limitations, and leveraging managed services for better efficiency.
Unbundling the Modern Streaming Stack With Dunith Dhanushka | Current 2022
- 1. Unbundling the Modern Streaming Stack Dunith Dhanushka - 05/10/2022 Navigating the Real-time Analytics Landscape
- 2. About Me twitter.com/dunithd medium.com/event-driven-utopia linkedin.com/in/dunithd/ • I’m Dunith Dhanushka • Big data solution architect -> DevRel • Blogs at eventdrivenutopia.com
- 3. Background • This talk is based on my blog that I published in April, 2022. • This talk has been updated with a few new things since then. • Enjoy!
- 4. Goal of the Talk What Are We Going To Talk About Today? Introduce you to the things required to build real-time applications that harness value from streaming data
- 5. The Plan The Order of Things 1. A refresher on streaming data 2. The classic streaming stack 3. The modern streaming stack 4. Current trends and the future outlook
- 6. What is a Streaming Stack?
- 7. Streaming Data What Is a stream? A stream is a continuous, never-ending data f low with no beginning or end. The data is incrementally made available over time, enabling you to act upon it without needing to be downloaded f irst.
- 8. Events Streams are made of events A data stream consists of a series of data points ordered in time. Each data point represents an “event” or a change in the state of the business. T4 T3 T2 T1 T0 Event source Event stream Time Events
- 9. Event First Thinking Modelling State Changes in Systems A user with ID 1234 purchased item 567 for $3.99 on 2022/06/12 at Austin, TX Fact Value User ID 1234 Item ID 567 Price Paid $3.99 Date 2022/06/12 Place Austin, TX • Events represents facts. • Events are immutable. • Events belong to the past.
- 10. Making Sense of Streaming Data
- 11. Events Have A Shelf Life Act Fast Before You Lose Their Value Image credit - https://d3i71xaburhd42.cloudfront.net/8cb6c2711afd3e504400ee12d3b582cc06348b08/7-Figure2-1.png
- 12. Real-time Analytics Extracting Value From Events As Soon as They Are Made Available REAL-TIME ANALYTICS Insights React Streams of Events
- 13. What is a Streaming Stack?
- 14. A streaming stack is the processes, tools, and technologies you use to derive insights from unbounded data.
- 15. The Classic Streaming Stack
- 16. The Beginning • Real-time analytics dates back to decades, existed in the forms of Complex Event Processing (CEP) and Event Stream Processing (ESP). • Most of the work has been academic. But few vendors like Progress Apama, Esper, Tibco, and Streambase tried bringing it to the mass market.
- 17. Then Came Big Data…
- 19. Lambda Architecture Promotes A Uni f ied Serving Layer Image credit - https://www.databricks.com/glossary/lambda-architecture
- 20. Why Didn’t It Pick Up?
- 22. • Overly complicated technology: Specialised skillset of distributed systems and performance engineering. • Limited only to the JVM: Non- JVM developers had no option rather than adapting. • Higher footprint on infrastructure: Stream processors tax heavily on the CPU and RAM. • Maintenance overhead: Having to maintain both speed and batch layers.
- 23. The Modern Streaming Stack
- 24. Modern Streaming Stack Modern Cloud-native tools Managed and Serverless platforms Rich tooling and developer experience Expressive programming model
- 25. MSS is the classic streaming stack reimagined with self-service cloud-native tools providing a simpli f ied yet powerful developer experience to build real-time analytics applications.
- 26. Modern Streaming Stack STREAMING DATA PLATFORM STREAM PROCESSING EVENT PRODUCERS TIERED STORAGE DATA API, METADATA & GOVERNANCE Data-driven Applications Operational Systems Real-time Analytics SERVING LAYER
- 27. The Unbundling
- 28. Event Production/Enablement The Origins of Events STREAMING DATA PLATFORM Language Speci f ic SDK Clients
- 30. • Ingest events from sources in a scalable manner, and store them durably until they are processed. • Based on an immutable, distributed log f ile. Events are appended to the log and partitioned across multiple servers for durability and scalability. EVENT PRODUCERS Streaming Data Platform TOPIC TOPIC TOPIC TOPIC TOPIC TOPIC
- 33. STREAM PROCESSING Event-driven Microservices Streaming ETL • Stream joins for enrichment • Filtering/routing/transforming streams • Data integration • Repartitioning streams (re-keying) Streaming Analytics • Stateful aggregations • Window operations • Materialising streams, stream-table duality • Actors • Reactive logic execution • Event-by-event processing, triggering side e ff ects
- 35. Serving Layer
- 36. INPUT TOPIC OUTPUT TOPIC Event Streaming Platform STREAM PROCESSING Serving Layer Events Streaming ingestion Real-time Insights Consumption Internal/user-facing Analytics Data Applications Recommendation Ad-hoc Exploration
- 37. Serving Layer Expectations • Serve queries with sub-second latency to provide a better user experience. • Support a throughput of hundreds of thousands of queries per second to serve an Internet-scale user base. • Ensure data freshness — serve analytics from data ingested a few seconds ago. • Run complex OLAP queries, supporting joins, aggregations, and f iltering on large data sets.
- 38. Serving Layer Technology Choices Key-value stores, NoSQL databases Real-time OLAP Databases
- 39. Tiered Storage
- 40. Serving Layer STREAMING DATA PLATFORM New Events Older Events Tiered Storage • Back f illing • Hydrating new applications • Experimentation (ad-hoc querying) • Archival/regulatory compliance • Training ML models O ff line Use Cases
- 41. Data APIs, Metadata, and Governance
- 42. Analytics must be democratised and accessible across the board… Image credits - https://www.datanami.com/2022/01/21/data-meshes-set-to-spread-in-2022/, https://www.con f luent.io/blog/how-to-build-a-data-mesh-using- event-streams/
- 43. Event Mesh EVENT CATALOG SCHEMA REGISTRY STREAMING API GRAPHQL API Serving Layer STREAM PROCESSOR EVENT STREAMING PLATFORM Decision makers Data applications Regulatory bodies Business partners Real-time Insights
- 44. Technology Choices Standards Schema Registries
- 46. Convergence of Stream Processing and Serving Layer Streaming databases takes the stateful stream processing to the next level. SaaS o ff errings Integrated serving layer Write logic with SQL Pluggable integrations A ff ordable Developer friendly Pay-as-you-go Less components to manage Integrated tooling Caters to non-JVM developers Self-serve
- 47. Rise of The Lakehouse Architecture A Lakehouse combines a data warehouse, data lake, and an event streaming platform together. High-throughput streaming ingestion Change Data Capture Upserts Transactions Table formats
- 48. Takeaways
- 49. Takeaways There’s No Silver Bullet • Start small, build the critical path, and iterate. • Pick components based on the need and know their limitations. • Experiment, fail fast, and fail cheap. • Go for managed services, if the team is small and new to streaming technologies. • Learn from mistakes, establish processes, and share wisdom!!
- 51. Thank you! twitter.com/dunithd medium.com/event-driven-utopia linkedin.com/in/dunithd/ Find me at: