Big Data LDN 2018: THE FUTURE OF STREAMING: GLOBAL APPS, EVENT STORES AND SERVERLESS
- 1. The Future of Streaming: Global Apps, Event Stores and Serverless Ben Stopford Office of the CTO, Confluent
- 2. Streaming sits at the intersection of how we deal with data and how we write programs
- 3. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
- 4. Apps Apps Apps Apps Search Monitoring Apps Apps Apps Apps Apps Apps Search Monitoring Apps Apps Apps Search NoSQL Apps Apps DWH Hado STREAM ING PLATFORM Apps Search NoSQL Apps DWH STREAMING PLATFORM PRODUCERCONSUMER Streaming Platform
- 5. Event Storage Kafka stores petabytes of data Stream Processing Real-time processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Roots in big data messaging
- 6. > 2 trillion messages per day
- 7. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
- 8. Events change our thinking Monolithic Approach -A database -a variable -a singleton -a RPC Event-First Approach - An event - A stream - A log - A stream processor
- 9. Event-driven programs have location transparency
- 10. They take us on journeys
- 11. Events let us run anywhere Interconnecting these separate worlds as real-time ecosystems
- 12. The future lies in integrated global streaming
- 13. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
- 14. Events change the way we observe the world around us
- 15. Events: A fact. An observation of the world. An payment A page view A log line A sensor reading
- 16. Events come in streams Apps M onitoring Security Apps Apps L A T F O R M Event Stream Order of events is important Apps Monitoring Apps Apps O R M
- 17. Events record what happened. Streams record how it happened
- 18. Traditional systems use mutable state DB This isn’t wrong, it’s just lossy Apps Search Mon Apps Apps S T R E A M I N G P L A T F O R M
- 19. Events record the user’s journey Shopping Cart Events 2 Trousers added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
- 20. Stored as a stream Stored statefully (think DB) 12.42 12.44 12.49 12.50 12.59 Information lost! Event User Journey
- 21. 12.42 12.44 12.49 12.50 12.59 We can derive the current state (but not the other way around) Apps Apps DERIVE Stream Processor
- 22. Streaming is a form of Event Sourcing The current state is a projection of the recording Familiar Stateful View LOSSY PROJECTION Stream = Exactly what happened
- 23. Streams let us “observe the game” one event at a time
- 24. The End State Often the game is more important than the end state The Game
- 25. A stock price: observe the game, not just the current state
- 26. A customer journey: observe everything
- 27. Formula 1
- 28. Formula 1: Observe the game, optimize the end state now and in the future End state
- 29. Formulae 1 – High-Level Architecture • 400 Sensors on car • 70,000 derivative measures • Events streamed back to base • Analyzed in real time • Tire modelling • Racing line • Aerodynamics • Machine Learning and Physics Models. • Replayed later for post race analysis. Race Track HQ e.g. Tire modelling: - Temp - Pressure - Suspension compression Stream Processing
- 31. ML SourceofTruth Retain events, rewind and replay the stream processor
- 32. Another form of “Event Sourcing” - Record what happened - Rewind, replay and rederive (View, App, ML, Physics Model etc.)
- 33. Billing Shipping Fraud Fraud CUSTOMER ANALYTICS EVENT STORE Rich, real-time recordings of customers and companies
- 34. Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres KSQL Other Kafka Select Organizational Events Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing Spark KSQL Orders History 1w All Event stores make data self service (real time & historical)
- 35. Rich recordings of customers and companies Real-time Historical Self Service
- 36. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
- 37. A future of Streaming changes how we observe the game. Cloud changes how we play it.
- 38. Apps Apps Apps Apps Search Monitoring Apps Apps Apps Apps Apps Apps Search Monitoring Apps Apps Apps Search NoSQL Apps Apps DWH Hado STREAM ING PLATFORM Apps Search NoSQL Apps DWH STREAMING PLATFORM PRODUCERCONSUMER Confluent Cloud 2019 2019
- 39. Serverless and Stream Processing are closely related
- 40. Using FaaS • Write a function • Upload • Configure a trigger (HTTP, Event, Object Store, Database, Timer etc.)
- 41. FaaS in a Nutshell • Short lived (max ~5 mins) • Pay as you use • 0-1000 concurrent functions, autoscales with load • Interesting for spikey compute • Interesting for low priority use cases e.g. CI systems.
- 42. But there are open questions
- 43. Serverless Developer Ecosystem • Runtime diagnostics • Monitoring • Deploy loop • Testing • IDE integration Currently quite poor
- 44. Harder than current approaches Easier than current approaches Amazon Google Microsoft
- 46. FaaS is event-driven But it isn’t streaming
- 47. Serverless Way: event driven but not streaming Orders Customers Payments FaaS FaaS FaaS
- 48. STREAMING: Event-first - how we think Event-sourced - how we store Event-driven - how we combine data and interact
- 49. Transaction Orders Payments KSQL Customers Streaming is Event-First, Event-Sourced & Event-Driven Stateful or Stateless
- 50. FaaSFaaSFaaS Transaction KSQL Stream processors can act as a “data layer” for FaaS ? FaaSFaaS StatelessStateful (slower elasticity) Orders Payments Customers
- 51. FaaSFaaSFaaS Transaction Orders Payments KSQL Customers StatelessStateful Inherit Kafka’s Rich Feature Set? FaaSFaaS
- 52. FaaS Traditional Application Event-Driven Application Application Database KSQL Stateful Data Layer FaaS FaaS FaaS FaaS FaaS Streaming Event-first Event-sourced Event-driven Stateless Stateless Stateless Compute Layer Auto-scaling, correctness, pluggability
- 53. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
- 54. GLOBAL SYSTEMS, STORED EVENTS, CLOUD NATIVE STREAM PROCESSING Data Layer FaaS FaaS FaaS FaaS FaaS