JavaOne2017: ACID Is So Yesterday: Maintaining Data Consistency with Sagas
10 likes1,413 views
The document presents a discussion on maintaining data consistency in microservices architecture using 'sagas' as an alternative to traditional ACID transactions. It outlines the challenges of distributed data management, highlighting the need for orchestration and choreography in coordinating sagas while emphasizing the importance of transactional messaging. Ultimately, it advocates for microservice designs that facilitate loose coupling and reliable transaction management.
JavaOne2017: ACID Is So Yesterday: Maintaining Data Consistency with Sagas
- 1. @crichardson ACID Is So Yesterday: Maintaining Data Consistency with Sagas Chris Richardson Founder of Eventuate.io Founder of the original CloudFoundry.com Author of POJOs in Action @crichardson chris@chrisrichardson.net http://eventuate.io
- 2. @crichardson Presentation goal Distributed data management challenges in a microservice architecture Sagas as the transaction model
- 4. @crichardson About Chris Consultant and trainer focusing on modern application architectures including microservices (http://www.chrisrichardson.net/)
- 5. @crichardson About Chris Founder of a startup that is creating an open-source/SaaS platform that simpliïŹes the development of transactional microservices (http://eventuate.io)
- 7. @crichardson Agenda ACID is not an option Overview of sagas Coordinating sagas Sagas and inter-service communication
- 8. The microservice architecture structures an application as a set of loosely coupled services
- 9. @crichardson Microservices enable continuous delivery/deployment Process: Continuous delivery/deployment Organization: Small, agile, autonomous, cross functional teams Architecture: Microservice architecture Enables Enables Enables Successful Software Development Services = testability and deployability Teams own services
- 12. @crichardson Loose coupling = encapsulated data Order Service Customer Service Order Database Customer Database Order table Customer table orderTotal creditLimit
- 13. @crichardson How to maintain data consistency?!?!? Invariant: sum(open order.total) <= customer.creditLimit
- 14. @crichardson Cannot use ACID transactions BEGIN TRANSACTION ⊠SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? ⊠SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? ⊠INSERT INTO ORDERS ⊠⊠COMMIT TRANSACTION Private to the Order Service Private to the Customer Service Distributed transactions
- 15. @crichardson 2PC is not an option Guarantees consistency BUT 2PC coordinator is a single point of failure Chatty: at least O(4n) messages, with retries O(n^2) Reduced throughput due to locks Not supported by many NoSQL databases (or message brokers) CAP theorem 2PC impacts availability âŠ.
- 17. @crichardson Agenda ACID is not an option Overview of sagas Coordinating sagas Sagas and inter-service communication
- 18. @crichardson From a 1987 paper
- 19. @crichardson Saga Use Sagas instead of 2PC Distributed transaction Service A Service B Service A Local transaction Service B Local transaction Service C Local transaction X Service C
- 20. @crichardson Order Service Create Order Saga Local transaction Order state=PENDING createOrder() Customer Service Local transaction Customer reserveCredit() Order Service Local transaction Order state=APPROVED approve order() createOrder()
- 21. @crichardson If only it were this easyâŠ
- 22. @crichardson Rollback using compensating transactions ACID transactions can simply rollback BUT Developer must write application logic to ârollbackâ eventually consistent transactions Careful design required!
- 23. @crichardson Saga: Every Ti has a Ci T1 T2 ⊠C1 C2 Compensating transactions T1 T2 C1 FAILS
- 24. @crichardson Order Service Create Order Saga - rollback Local transaction Order createOrder() Customer Service Local transaction Customer reserveCredit() Order Service Local transaction Order reject order() createOrder() FAIL InsufïŹcient credit
- 25. @crichardson Sagas complicate API design Synchronous API vs Asynchronous Saga Request initiates the saga. When to send back the response? Option #1: Send response when saga completes: + Response speciïŹes the outcome - Reduced availability Option #2: Send response immediately after creating the saga (recommended): + Improved availability - Response does not specify the outcome. Client must poll or be notiïŹed
- 26. @crichardson Revised Create Order API createOrder() returns id of newly created order NOT fully validated getOrder(id) Called periodically by client to get outcome of validation
- 27. @crichardson Minimal impact on UI UI hides asynchronous API from the user Saga will usually appear instantaneous (<= 100ms) If it takes longer UI displays âprocessingâ popup Server can push notiïŹcation to UI
- 28. @crichardson Interwoven sagas complicate business logic Order Service Local transaction Order state=PENDING createOrder() Customer Service Local transaction Customer reserveCredit() Order Service Local transaction cancelOrder() ?
- 29. @crichardson How handle interwoven sagas? What does it mean to cancel a PENDING Order? âInterruptâ the Create Order saga? Wait for the Create Order saga to complete?
- 30. @crichardson Agenda ACID is not an option Overview of sagas Coordinating sagas Sagas and inter-service communication
- 31. @crichardson How to sequence the saga transactions? After the completion of transaction Ti âsomethingâ must decide what step to execute next Success: which T(i+1) - branching Failure: C(i - 1)
- 32. @crichardson Choreography: distributed decision making vs. Orchestration: centralized decision making
- 33. @crichardson Option #1: Choreography-based coordination using events Order Service Customer Service Order created Credit Reserved Credit Limit Exceeded Create Order OR Customer creditLimit creditReservations ... Order state total ⊠create() reserveCredit() approve()/reject()
- 34. BeneïŹts and drawbacks of choreography BeneïŹts Simple, especially when using event sourcing Participants are loosely coupled Drawbacks Cyclic dependencies - services listen to each otherâs events Overloads domain objects, e.g. Order and Customer know too much Events = indirect way to make something happen
- 35. @crichardson Order Service Option #2: Orchestration-based saga coordination Local transaction Order state=PENDING createOrder() Customer Service Local transaction Customer reserveCredit() Order Service Local transaction Order state=APPROVED approve order() createOrder() CreateOrderSaga
- 36. @crichardson A saga (orchestrator) is a persistent object that tracks the state of the saga and invokes the participants
- 37. @crichardson Saga behavior On create: Invokes a saga participant On reply: Determine which saga participant to invoke next Invokes saga participant Updates its state âŠ
- 38. @crichardson Order Service CreateOrderSaga orchestrator Customer Service Create Order Customer creditLimit creditReservations ... Order state total⊠reserveCredit() CreateOrder Saga OrderService create() create() approve() creditReserved()
- 39. @crichardson CreateOrderSaga deïŹnition Sequence of steps step = (Ti, Ci) Build command to send Sagaâs Data
- 40. @crichardson Customer Service command handler Route command to handler Reserve credit Make reply message
- 41. @crichardson Coming next week to https://github.com/eventuate-tram
- 42. BeneïŹts and drawbacks of orchestration BeneïŹts Centralized coordination logic is easier to understand Reduced coupling, e.g. Customer knows less Reduces cyclic dependencies Drawbacks Risk of smart sagas directing dumb services
- 43. @crichardson Agenda ACID is not an option Overview of sagas Coordinating sagas Sagas and inter-service communication
- 44. @crichardson Saga Participant About Saga orchestrator participant communication Saga Orchestrator Saga Participant command reply Saga must complete even if there are transient failures
- 45. @crichardson Use asynchronous messaging Ensures sagas complete when participants are temporarily unavailable
- 46. @crichardson Create Order Saga - messaging Order Service Create Order Saga Message Broker Customer Service Customer Customer Request Channel Saga Reply Channel Reserve Credit Reserve Credit Reply
- 47. @crichardson Messaging must be transactional Service Database Message Broker update publish How to make atomic without 2PC?
- 48. @crichardson Option #1: Use database table as a message queue ACID transaction See BASE: An Acid Alternative, http://bit.ly/ebaybase DELETE Customer Service ORDER_ID CUSTOMER_ID TOTAL 99 CUSTOMER_CREDIT_RESERVATIONS table 101 1234 ID TYPE DATA DESTINATION MESSAGE table 84784 OrderCreated {âŠ} ⊠INSERT INSERT Message Publisher QUERY Message Broker Publish Local transaction ?
- 49. @crichardson Publishing messages Poll the MESSAGE table OR Tail the database transaction log
- 50. @crichardson Option #2: Event sourcing: event-centric persistence Service Event Store save events and publish Event table Entity type Event id Entity id Event data Order 902101 âŠOrderApproved Order 903101 âŠOrderShipped Event type Order 901101 âŠOrderCreated Every state change event
- 51. @crichardson Summary Microservices tackle complexity and accelerate development Database per service is essential for loose coupling Use sagas to maintain data consistency across services Use transactional messaging to make sagas reliable