Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)
21 likes54,646 views
The document discusses how to achieve eventual consistency in JVM microservices through event sourcing, addressing issues related to transactions and queries. It outlines the advantages and challenges of using event sourcing, such as improved audit capabilities and potential complexities in querying. The presentation also emphasizes the need for Command Query Responsibility Segregation (CQRS) to efficiently handle queries in an event-driven microservice architecture.
Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)
- 1. @crichardson Handling Eventual Consistency in JVM Microservices with Event Sourcing Chris Richardson Founder of Eventuate.io @crichardson http://eventuate.io Copyright © 2015. Chris Richardson Consulting, Inc. All rights reserved Kenny Bastani Spring Developer Advocate @kennybastani http://pivotal.io
- 2. @crichardson Presentation goal Show how event sourcing is a great foundation for a microservice architecture
- 5. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
- 6. @crichardson Traditional persistence Order id state …. 101 ACCEPTED Order table … ID STATE … 102 … …
- 7. @crichardson But how did we get here? Who did what and when? What was the state of the Order last Monday at 3:01pm?
- 10. @crichardson Usually auditing, history and temporal queries is additional code and/or an after-thought
- 11. @crichardson Event sourcing For each domain object (i.e. DDD aggregate): Identify (state changing) domain events, e.g. use Event Storming Define Event classes For example, Order events: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped
- 12. @crichardson Persists events NOT current state Event table Entity type Event id Entity id Event data Order 902101 …OrderApproved Order 903101 …OrderShipped Event type Order 901101 …OrderCreated
- 13. @crichardson Replay events to recreate state Order state OrderCreated(…) OrderAccepted(…) OrderShipped(…) Events Periodically snapshot to avoid loading all events
- 14. @crichardson Benefits of event sourcing Reifies state changes: Built in, reliable audit log temporal queries Preserved history More easily implement future requirements Eliminates O/R mapping problem (mostly) Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,…
- 15. @crichardson Drawbacks of event sourcing Requires application rewrite Weird and unfamiliar style of programming Events live forever carefully evolve schema Querying the event store can be challenging Current state is no longer directly available Often need to maintain views for efficiency
- 17. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
- 18. @crichardson The Microservice architecture tackles complexity through modularization
- 19. @crichardson Microservice architecture Browser Mobile Device Store Front UI API Gateway Catalog Service Review Service Order Service … Service Catalog Database Review Database Order Database … Database HTML REST REST Apply X-axis and Z-axis scaling to each service independently
- 20. @crichardson But there are challenges implementing transactions and queries
- 21. @crichardson ACID transactions cannot be used BEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION Private to the Order Service Private to the Customer Service Requires 2PC
- 22. @crichardson 2PC is not a viable option Guarantees consistency BUT 2PC is best avoided Not supported by many NoSQL databases etc. CAP theorem 2PC impacts availability ….
- 23. @crichardson Queries can’t use joins SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ? Customer Service Order Service
- 24. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
- 26. @crichardson Order Management Order id : 4567 total: 343 state = CREATED Customer Management Customer creditLimit : 12000 creditReservations: {} Customer creditLimit : 12000 creditReservations: { 4567 -> 343} Order id : 4567 total: 343 state = APPROVED Eventually consistent credit checking Message Bus createOrder() Publishes: Subscribes to: Subscribes to: publishes: OrderCreatedEvent CreditReservedEvent OrderCreatedEvent CreditReservedEvent
- 27. @crichardson How atomically update database and publish an event Order Service Order Database Message Broker insert Order publish OrderCreatedEvent dual write problem ?
- 28. @crichardson Failure = inconsistent system Order Service Order Database Message Broker insert Order publish OrderCreatedEvent X
- 29. @crichardson 2PC is not an option
- 30. @crichardson How to reliably publish events when state changes?
- 31. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
- 32. @crichardson Event sourcing = event-centric persistence Application Database Event store update publish X
- 33. @crichardson Event Store Application architecture Order 123 Customer 456 OrderCreated OrderApproved … CustomerCreated CustomerCreditReserved … CreateOrder UpdateOrder GetOrder Subscribe Order Service CreateCustomer UpdateCustomer GetCustomer Subscribe Customer Service
- 34. @crichardson Request handling in an event sourced application HTTP Handler Event Store pastEvents = findEvents(entityId) Order new() applyEvents(pastEvents) newEvents = processCmd(someCmd) saveEvents(newEvents) (optimistic locking) Order Service applyEvents(newEvents)
- 35. @crichardson Event Store publishes events consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) Customer update() Customer Service
- 36. @crichardson Event Store publishes events consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) CQRS View update() Service Xyz send notifications …
- 37. Event store = database + message broker Hybrid database and message broker Implementations: Home grown/DIY geteventstore.com by Greg Young http://eventuate.io (mine) Event Store Save aggregate events Get aggregate events Subscribe to events
- 38. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
- 39. @crichardson Find recent, valuable customers SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ? Customer Service Order Service What if event sourcing is used?…. is no longer easy
- 40. @crichardson Use Command Query Responsibility Segregation (CQRS)
- 41. @crichardson Command Query Responsibility Segregation (CQRS) Application logic Commands Queries X POST PUT DELETE GET
- 42. @crichardson Command Query Responsibility Segregation (CQRS) Command side Commands Event Sourcing domain objects Event Store Events Query side Queries (Materialized) View Events POST PUT DELETE GET
- 43. @crichardson Query side design Event Store Updater View Updater Service Events Reader HTTP GET Request View Query Service View Store e.g. MongoDB ElasticSearch Neo4J update query
- 44. @crichardson Persisting a customer and order history in MongoDB { "_id" : "0000014f9a45004b 0a00270000000000", "_class" : "net.chrisrichardson…..views.orderhistory.CustomerView", "version" : NumberLong(5), "orders" : { "0000014f9a450063 0a00270000000000" : { "state" : "APPROVED", "orderId" : "0000014f9a450063 0a00270000000000", "orderTotal" : { "amount" : "1234" } }, "0000014f9a450063 0a00270000000001" : { "state" : "REJECTED", "orderId" : "0000014f9a450063 0a00270000000001", "orderTotal" : { "amount" : "3000" } } }, "name" : "Fred", "creditLimit" : { "amount" : "2000" } } Denormalized = efficient lookup
- 45. @crichardson Persisting customers and order info using Spring Data for MongoDB...
- 46. @crichardson Persisting customers and order using Spring Data for MongoDB...
- 47. Benefits and drawbacks of CQRS Benefits Necessary in an event sourced architecture Separation of concerns = simpler command and query models Supports multiple denormalized views Improved scalability and performance Drawbacks Complexity Potential code duplication Replication lag/eventually consistent views
- 49. @crichardson Summary Microservice architecture functionally decomposes an application into services Transactions and queries resist decomposition Use an event-driven architecture based on event sourcing to maintain data consistency Implement queries using CQRS