Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"
4 likes1,531 views
The document discusses the convergence of stream processing and microservice architecture, highlighting the evolution and requirements of both technologies. It emphasizes the role of evented services and the potential to derive microservices from data streams for improved scalability, resilience, and upgradeability. Finally, it introduces Lightbend's platforms that support these innovations, encouraging the use of appropriate tools for each component in a streaming pipeline.
Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"
- 1. The convergence of stream processing & microservice architecture √Deputy CTO, Lightbend @viktorklang / v@lightbend.com / viktorklang.com
- 3. Overview
- 4. History of Data Processing 4 Offline Batch processing 4 MapReduce 4 Lambda Architecture
- 5. History of Services 4 Client-Server 4 CORBA & DCOM 4 EJB & N-Tier 4 SOA & ESB 4 HTTP-API & REST
- 6. Today
- 7. Catch of the day Live1 Stream Processing & Microservice Architecture 1 Near real-time
- 8. Shared requirements: 4 Available 4 Elastically scalable 4 Resilient & self-healing 4 Loosely coupled & upgradeable
- 9. Reactive Systems2 Available (responsive), under failure (resilience) and under load (elasticity), by being message-driven. 2 https://www.reactivemanifesto.org/
- 10. Reactive Streams3 Provides a standard for asynchronous stream processing with non-blocking back pressure. 3 www.reactive-streams.com & java.util.concurrent.Flow (JDK9+)
- 11. Ingredients 4 Stream processing stages: Stateless & Stateful 4 Microservices: RPC-style & Evented 4 Domain Events with Schema/Codec
- 12. #define Events4 4 Immutable recordings of observations 4 Can be disregarded but not be retracted once accepted 4 Can be invalidated but (typically) not be deleted 4 Knowledge accrues with every processed event 4 Learn more: https://speakerdeck.com/jboner/designing-events-first-microservices
- 14. Realizationevented services are stream processing stages
- 16. Is this the solution? 4 Alice: So we we lift evented services into the stream processing pipeline … 4 Bob: … and then we're done! 4 <record scratch> 4 Narrator: They're forgetting the request-reply use- case which RPC services excel at.
- 17. Services (rpc)
- 18. The improved solution 4 lift the service into a stream stage 4 lift input Events into Commands and submit 4 capture responses & convert into output Events
- 19. What if we could derive microservices from data streams?
- 23. Unification 4 Microservices are a part of a streaming pipeline 4 A pipeline can now be exposed as Microservices 4 We can independently upgrade parts of the pipeline
- 24. Developing converged applications 4 Deployment Orchestration 4 Events-First DDD & EventStorming 4 (Meta)Data-driven Continuous Deployment 4 Use of the right tools for each component
- 25. Beyond
- 27. End of stream
- 28. Explore what we do at Lightbend: 4 Lightbend Fast Data Platform 4 https://www.lightbend.com/products/fast-data- platform 4 Lightbend Reactive Platform 4 https://www.lightbend.com/products/reactive-platform 4 Lightbend Enterprise Suite 4 https://www.lightbend.com/products/enterprise-suite
- 29. Thank you for attending! The convergence of stream processing & microservice architecture √ Deputy CTO, Lightbend @viktorklang / v@lightbend.com / viktorklang.com