Event driven microservices
- 1. Event-driven microservices using Apache Kafka Andrew Schofield Chief Architect, Event Streams IBM Hursley Park © 2019 IBM Corporation
- 2. What and why microservices? Microservices is a technique for structuring an application as a collection of services: • Self-contained with clear interfaces and a distinct purpose • Loosely coupled – communicate over a network • Independently deployable, scalable, maintainable and testable © 2019 IBM Corporation Microservices ApplicationMonolithic Application
- 3. Being event-driven is different © 2019 IBM Corporation Source: Gartner - From data-centric to event-centric IT priority
- 4. Why event-driven? • Proper loose coupling means asynchronous communication • Enables responsive applications • In the real world, things often take time to complete © 2019 IBM Corporation
- 5. Event-driven microservices • Microservices communicate primarily using events, with APIs where required • Microservices can produce and consume events using the publish/subscribe pattern • Events are handled by an event backbone • Data is eventually consistent © 2019 IBM Corporation Microservice Microservice Microservice Microservice Microservice Event backbone Publish Service discoveryMicroservice Microservice Microservice Subscribe Subscribe API API Publish JSON / HTTP JSON / HTTP Microservices application
- 6. Apache Kafka is an Open-Source Streaming Platform © 2019 IBM Corporation PUBLISH & SUBSCRIBE Read and write streams of events, like a traditional messaging system. PROCESS Support scalable stream processing applications that react to events in real-time. STORE Store streams of data safely in a distributed, replicated, fault-tolerant cluster. https://kafka.apache.org/ Kafka Cluster Producer ProducerProducer Consumer ConsumerConsumer Stream Processor Stream Processor Source Connector Sink Connector
- 7. Event-driven terminology Event A message usually representing a fact at a particular time Event stream A continuous, ordered, unbounded sequence of events Stream processing Computation optimized for continuous processing, often with state Stream history The historical sequence of the events on an event stream © 2019 IBM Corporation
- 8. Apache Kafka as the event backbone © 2019 IBM Corporation
- 9. Apache Kafka as the event backbone © 2019 IBM Corporation Microservice Microservice
- 10. Apache Kafka as the event backbone © 2019 IBM Corporation 1 Building Blocks 1 :: Event sources Microservice Microservice
- 11. Apache Kafka as the event backbone © 2019 IBM Corporation 1 2 Building Blocks 1 :: Event sources 2 :: Stream processing Microservice Microservice
- 12. Apache Kafka as the event backbone © 2019 IBM Corporation 1 2 App 3 Building Blocks 1 :: Event sources 2 :: Stream processing 3 :: Event archive Microservice Microservice
- 13. Apache Kafka as the event backbone © 2019 IBM Corporation App 1 2 3 4 Building Blocks 1 :: Event sources 2 :: Stream processing 3 :: Event archive 4 :: Notifications Microservice Microservice
- 14. How to get started • You could run an event storming workshop https://www.ibm.com/cloud/garage/architectures/eventDrivenArchitecture/event-storming-methodology • Identify: • Events • Actors • Data • Commands © 2019 IBM Corporation
- 15. A simple stream processing example © 2019 IBM Corporation Event backbone Transactions
- 16. A simple stream processing example © 2019 IBM Corporation Event backbone Transactions Alerts
- 17. A simple stream processing example © 2019 IBM Corporation Event backbone Transactions Alerts SMS alerts Email alerts Push alerts
- 18. A simple stream processing example © 2019 IBM Corporation Event backbone Send SMS Transactions Alerts Send emails Send push notifications SMS alerts Email alerts Push alerts
- 19. © 2019 IBM Corporation A more complex event-driven example https://ibm-cloud-architecture.github.io/refarch-kc/design/readme/
- 20. A more complex example – component view © 2019 IBM Corporation Fleet microservice Containers microservice Voyages microservice Orders microservice APIAPI
- 21. A more complex example – component view © 2019 IBM Corporation Fleet microservice Containers microservice Voyages microservice Orders microservice APIAPI
- 22. A more complex example – component view © 2019 IBM Corporation Event backbone ships containers orders voyages Fleet microservice Containers microservice Voyages microservice Orders microservice APIAPI
- 23. A more complex example – component view © 2019 IBM Corporation Event backbone ships containers problems orders voyages Fleet microservice Containers microservice Voyages microservice Orders microservice Streaming analytics APIAPI
- 24. Patterns for event-driven microservices • You want: • Loose coupling • Data consistency • Efficient queries © 2019 IBM Corporation
- 25. Patterns for event-driven microservices • You want: • Loose coupling • Data consistency • Efficient queries • You don’t want: • Chaos © 2019 IBM Corporation
- 26. Patterns for event-driven microservices • You want: • Loose coupling • Data consistency • Efficient queries • You don’t want: • Chaos • You need PATTERNS © 2019 IBM Corporation
- 27. Pattern – Database per service • Each microservice persists its own data + Protects independence of the microservice against external change - Introduces complexity for data consistency across microservices Event backbone ships containers problems orders voyages Fleet microservice Containers microservice Voyages microservice Orders microservice © 2019 IBM Corporation
- 28. Kafka log compaction for data replication © 2019 IBM Corporation 0 key:a val:A1 1 key:b val:B1 2 key:a val:A2 3 key:c val:C1 Partition 0 4 key:c val:C2 5 key:b val: Producer writes Consumer takes latest values and builds own data store reads Key Value a A2 b <deleted> c C2 Old New Source change log Read
- 29. Kafka log compaction for data replication © 2019 IBM Corporation 0 key:a val:A1 1 key:b val:B1 2 key:a val:A2 3 key:c val:C1 Partition 0 4 key:c val:C2 5 key:b val: 2 key:a val:A2 4 key:c val:C2 5 key:b val: Periodic compaction eliminates duplicate keys to minimize storage Partition 0 (rewritten) Key Value a A2 b <deleted> c C2
- 30. Pattern – Event sourcing • Every state change is captured as an event stored in sequence + Publishing events is atomic - Event store is hard to query efficiently © 2019 IBM Corporation OrderCreated OrderUpdated OrderUpdated OrderConfirmed OrderShipped OrderDelivered OrderInTransit Orders microservice Orders DB Command handler USER Time OrderPending OrderBooked OrderAssigned
- 31. Pattern – Sagas • Orchestration of multi-step operations across microservices + Data consistency across microservices without distributed transactions - Programming can be complex, particularly for failure compensation © 2019 IBM Corporation Event backbone Fleet microservice Containers microservice Voyages microservice Orders microservice 1 2 3 4 5
- 32. Pattern – CQRS • Command Query Responsibility Segregation + Atomic writes and efficient reads at the same time - Complex to program correctly © 2019 IBM Corporation Event backbone 1 2 3 4 5 Fleet MS WRITE MODEL Fleet MS READ MODEL Write-optimized store Read-optimized store Write API Read API
- 33. Summary • Event-driven microservices offer an effective way to build loosely- coupled applications • Techniques such as event storming can be used to derive the objects, commands and events in an application • Patterns such as event sourcing and sagas make the complexity manageable © 2019 IBM Corporation