Developing microservices with aggregates (melbourne)
0 likes940 views
The document discusses using domain-driven design aggregates and event sourcing to develop microservices architectures. It presents an example of using aggregates to partition a customer and order domain across microservices. It also discusses using event-driven sagas to maintain consistency between aggregates instead of two-phase commit transactions. Maintaining state as events allows reconstructing state from history and reliably publishing events.
Developing microservices with aggregates (melbourne)
- 1. @crichardson Developing Microservices with Aggregates Chris Richardson Founder of Eventuate.io Founder of the original CloudFoundry.com Author of POJOs in Action @crichardson chris@chrisrichardson.net http://eventuate.io
- 2. @crichardson Presentation goal Show how Domain Driven Design Aggregates and Microservices are a perfect match
- 4. @crichardson About Chris Consultant and trainer focusing on modern application architectures including microservices (http://www.chrisrichardson.net/)
- 5. @crichardson About Chris Founder of a startup that is creating an open-source/SaaS platform that simplifies the development of transactional microservices (http://eventuate.io)
- 7. @crichardson Agenda The problem with Domain Models and microservices Overview of aggregates Maintaining consistency between aggregates Using event sourcing with Aggregates Customers and Orders example Using explicit saga orchestration
- 8. @crichardson The Microservice architecture tackles complexity through modularization
- 11. @crichardson But there are challenges…
- 12. @crichardson Product service Customer serviceOrder service Domain model = tangled web of classes Order OrderLine Item quantity … Address street city … Customer Product name price ? ? creditLimit
- 13. @crichardson Reliance on ACID transactions to enforce invariants BEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION Simple and ACID
- 14. @crichardson But it violates encapsulation… BEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION Private to the Order Service Private to the Customer Service
- 15. @crichardson .. and requires 2PC BEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION
- 16. @crichardson 2PC is not an option Guarantees consistency BUT 2PC coordinator is a single point of failure Chatty: at least O(4n) messages, with retries O(n^2) Reduced throughput due to locks Not supported by many NoSQL databases (or message brokers) CAP theorem 2PC impacts availability ….
- 17. @crichardson Doesn’t fit with the NoSQL DB “transaction” model
- 18. @crichardson Agenda The problem with Domain Models and microservices Overview of aggregates Maintaining consistency between aggregates Using event sourcing with Aggregates Customers and Orders example Using explicit saga orchestration
- 19. @crichardson Domain Driven Design - building blocks Entity Value object Services Repositories Aggregates Adopted Ignored (at least by me)
- 20. About Aggregates Cluster of objects that can be treated as a unit Graph consisting of a root entity and one or more other entities and value objects Typically business entities are Aggregates, e.g. customer, Account, Order, Product, …. Order OrderLine Item quantity productId productName productPrice customerId Address street city …
- 21. Aggregate: rule #1 Reference other aggregate roots via identity (primary key) Order OrderLine Item quantity productId productName productPrice customerId Address street city …
- 22. @crichardson Foreign keys in a Domain Model?!?
- 23. @crichardson Domain model = collection of loosely connected aggregates Order OrderLine Item quantity … Address street city … Customer Product name price
- 24. @crichardson Product service Customer serviceOrder service Easily partition into microservices Order OrderLine Item quantity … Address street city … Customer Product name price
- 25. @crichardson Aggregate rule #2 Transaction = processing one command by one aggregate
- 26. @crichardson Product service Customer serviceOrder service Transaction scope = service Order OrderLine Item quantity … Address street city … Customer Product name price
- 28. @crichardson Agenda The problem with Domain Models and microservices Overview of aggregates Maintaining consistency between aggregates Using event sourcing with Aggregates Customers and Orders example Using explicit saga orchestration
- 29. @crichardson Customer management How to maintain data consistency between aggregates? Order management Order Service placeOrder() Customer Service updateCreditLimit() Customer creditLimit ... has ordersbelongs toOrder total Invariant: sum(open order.total) <= customer.creditLimit ?
- 30. @crichardson Saga Using event-driven Sagas instead of 2PC Distributed transaction Order Customer Local transaction Order Local transaction Customer Local transaction Order Event Event
- 31. @crichardson Order Service Saga-based, eventually consistent order processing Customer Service Order created Credit Reserved Credit Check Failed Create Order OR Customer creditLimit creditReservations ... Order state total … approve()/reject() Event Handler Event Handler reserveCredit()
- 32. @crichardson Order Service Order id : 4567 total: 343 state = CREATED Customer Service Customer creditLimit : 12000 creditReservations: {} Customer creditLimit : 12000 creditReservations: { 4567 -> 343} Order id : 4567 total: 343 state = APPROVED Eventually consistent credit checking Message Bus createOrder() Publishes: Subscribes to: Subscribes to: publishes: OrderCreatedEvent CreditReservedEvent OrderCreatedEvent CreditReservedEvent
- 33. @crichardson Complexity of compensating transactions ACID transactions can simply rollback BUT Developer must write application logic to “rollback” eventually consistent transactions For example: CreditCheckFailed => cancel Order Money transfer destination account closed => credit source account Careful design required!
- 34. @crichardson How atomically update database and publish an event Order Service Order Database Message Broker insert Order publish OrderCreatedEvent dual write problem ?
- 35. @crichardson Failure = inconsistent system Order Service Order Database Message Broker insert Order publish OrderCreatedEvent X
- 36. @crichardson 2PC is not an option
- 37. @crichardson Reliably publish events when state changes
- 38. @crichardson Agenda The problem with Domain Models and microservices Overview of aggregates Maintaining consistency between aggregates Using event sourcing with Aggregates Customers and Orders example Using explicit saga orchestration
- 39. @crichardson Event sourcing = event-centric persistence Application Database Event store update publish X
- 40. @crichardson Event sourcing For each DDD aggregate: Identify (state changing) domain events e.g. use Event Storming Define Event classes For example, Order events: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped
- 41. @crichardson Persists events NOT current state Order status …. 101 ACCEPTED Order table X
- 43. @crichardson Persists events NOT current state Event table Entity type Event id Entity id Event data Order 902101 …OrderApproved Order 903101 …OrderShipped Event type Order 901101 …OrderCreated
- 44. @crichardson Replay events to recreate state Order state OrderCreated(…) OrderAccepted(…) OrderShipped(…) Events Periodically snapshot to avoid loading all events
- 45. @crichardson The present is a fold over history currentState = foldl(applyEvent, initialState, events)
- 46. @crichardson Event Store Application architecture Order 123 Customer 456 OrderCreated OrderApproved … CustomerCreated CustomerCreditReserved … CreateOrder UpdateOrder GetOrder Subscribe Order Service CreateCustomer UpdateCustomer GetCustomer Subscribe Customer Service
- 47. @crichardson Request handling in an event sourced application HTTP Handler Event Store pastEvents = findEvents(orderId) Order new() applyEvents(pastEvents) newEvents = processCmd(someCmd) saveEvents(newEvents) (optimistic locking) Order Service applyEvents(newEvents)
- 48. @crichardson Event Store publishes events consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) Customer update() Customer Service
- 49. @crichardson Event Store publishes events consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) CQRS View update() Service Xyz send notifications …
- 50. Event store = database + message broker Home grown/DIY .NET geteventstore.com https:// www.lightbend.com/ lagom http://eventuate.io (mine) … Event Store Save aggregate events Get aggregate events Subscribe to events
- 51. @crichardson Benefits of event sourcing Solves data consistency issues in a Microservice/NoSQL based architecture Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,… Eliminates O/R mapping problem (mostly) Reifies state changes: Built in, reliable audit log temporal queries Preserved history More easily implement future requirements
- 52. @crichardson Drawbacks of event sourcing… Requires application rewrite Weird and unfamiliar style of programming Events = a historical record of your bad design decisions Must detect and ignore duplicate events Idempotent event handlers Track most recent event and ignore older ones …
- 53. @crichardson … Drawbacks of event sourcing Querying the event store can be challenging Some queries might be complex/inefficient, e.g. accounts with a balance > X Event store might only support lookup of events by entity id Must use Command Query Responsibility Segregation (CQRS) to handle queries application must handle eventually consistent data
- 54. @crichardson Agenda The problem with Domain Models and microservices Overview of aggregates Maintaining consistency between aggregates Using event sourcing with Aggregates Customers and Orders example Using explicit saga orchestration
- 55. @crichardson Orders and Customers example Kafka Rest API Event Store Event Store Adapter Customer Service Rest API Order Service Customer Aggregate Order Aggregate Rest API Order History Service MongoDB CQRS View Event Store Adapter Event Store Adapter Used to notify Order Service of invalid customer ids!
- 56. @crichardson Customer microservice Customer Service Customer Controller Spring Cloud Stream Kafka Customer Aggregate Rest API Customer Event Handlers Event Store Event Store Adapter Order*Event POST /customers
- 57. @crichardson The Customer aggregate Money creditLimit Map<OrderId, Money> creditReservations Customer List<Event> process(CreateCustomerCommand cmd) { … } List<Event> process(ReserveCreditCommand cmd) { … } … void apply(CustomerCreatedEvent anEvent) { … } void apply(CreditReservedEvent anEvent) { … } … State Behavior
- 58. @crichardson Familiar concepts restructured class Customer { public void reserveCredit( orderId : String, amount : Money) { // verify // update state this.xyz = … } public List<Event> process( ReserveCreditCommand cmd) { // verify … return singletonList( new CreditReservedEvent(…); ) } public void apply( CreditReservedEvent event) { // update state this.xyz = event.xyz }
- 62. @crichardson Creating a Customer save() concisely specifies: 1.Creates Customer aggregate 2.Processes command 3.Applies events 4.Persists events
- 63. @crichardson Event handling in Customers Durable subscription name 1. Load Customer aggregate 2. Processes command 3. Applies events 4. Persists events Triggers BeanPostProcessor Publish message to Spring Cloud Stream (Kafka) when Customer not found
- 64. @crichardson Event handler in Orders
- 65. @crichardson Agenda The problem with Domain Models and microservices Overview of aggregates Maintaining consistency between aggregates Using event sourcing with Aggregates Customers and Orders example Using explicit saga orchestration
- 66. @crichardson Order Service Order Service ⬄ Customer Service: tightly coupled, cyclic dependency 😢 Customer Service Order created Credit Reserved Credit Check Failed OR Must subscribe to all Order events that affect available credit! Order …
- 67. @crichardson Order Service Decouple by maintaining available credit in OrderService Customer Service Available Credit Changed Customer Created Customer creditLimit availableCredit ... Order state total … AvailableCredit Tracker creditLimit availableCredit ... Simpler 😄 Still cyclic 😢
- 68. Use an explicit saga Saga object that orchestrates the flow Created in response to an initial event, e.g. OrderCreated A state machine Receives events from aggregates Sends commands to aggregates Implemented as an event-sourced aggregate Saga Event Command
- 69. @crichardson Customer Service Order Service Using an OrderCreationSaga Order state total … AvailableCredit Tracker creditLimit availableCredit ... Order Creation Saga create() create() Reserve Credit Credit Reserved Approve Order Customer Update Available Credit OrderCreated Acyclic 😄
- 70. @crichardson Order saga state machine NEW SUCCEEDED FAILED OrderCreatedEvent / reserveCredit() CustomerCreditReservedEvent / approveOrder() EntityNotFoundCommandExecutionError CustomerCreditLimitExceededEvent / rejectOrder()
- 71. @crichardson Saga as a singleton state machine State + Data (State, Data, Event) (Commands, State’, Data’)
- 72. @crichardson (OrderSagaState, OrderSagaData, Event) (Commands, OrderSagaState’, OrderSagaData’)
- 74. @crichardson Initial action Starting action Triggers saga creation 1. Send command 2. Transition to new state 3. Create data
- 75. @crichardson Event handler Current state Triggering event 1. Send command 2. Transition to new state
- 76. @crichardson Kafka-based command bus Order Saga Credit Tracking Aggregate CreditTrackingCommand Topic OrderSagaCommandFailure Topic Reserve Credit CustomerNotFound Per aggregate type Per saga type
- 77. @crichardson Summary Aggregates are the building blocks of microservices Use event-driven sagas to maintain consistency between aggregates Event sourcing is a good way to implement a event-driven architecture Use explicit sagas to orchestrate transactions