Solving distributed data management problems in a microservice architecture (sfmicroservices)
22 likes8,735 views
The document discusses challenges of maintaining data consistency and performing queries in a microservices architecture. It recommends using sagas and compensating transactions to maintain consistency across services with private databases. For queries, it recommends API composition when possible, but for more complex queries, using the Command Query Responsibility Segregation (CQRS) pattern with separate read models optimized for different query types. The talk also covers event sourcing to provide reliable saga execution.
Solving distributed data management problems in a microservice architecture (sfmicroservices)
- 1. @crichardson Transactions and queries in a microservice architecture Chris Richardson Founder of Eventuate.io Founder of the original CloudFoundry.com Author of POJOs in Action @crichardson chris@chrisrichardson.net http://microservices.io http://eventuate.io http://plainoldobjects.com Copyright © 2017. Chris Richardson Consulting, Inc. All rights reserved
- 2. @crichardson Presentation goal How to solve distributed data management challenges in a microservice architecture
- 5. @crichardson About Chris Consultant and trainer focusing on modern application architectures including microservices (http://www.chrisrichardson.net/)
- 6. @crichardson About Chris Founder of a startup that is creating an open-source/SaaS platform that simplifies the development of transactional microservices (http://eventuate.io)
- 8. @crichardson Agenda Distributed data challenges in a microservice architecture Maintaining data consistency using sagas Implementing queries using API composition and CQRS
- 9. The microservice architecture is an architectural style that structures an application as a set of loosely coupled, services organized around business capabilities
- 10. @crichardson A well-designed microservice Meaningful business functionality Developed by a small team Minimal lead time/high deployment frequency
- 11. @crichardson Goal: enable continuous delivery of complex applications
- 12. @crichardson Goal: enable continuous delivery of complex applications Process: Continuous delivery/deployment Organization: Small, agile, autonomous, cross functional teams Architecture: Microservice architecture Enables Enables Enables Successful Software Development Services = testability and deployability Team service(s)
- 13. @crichardson Microservice architecture Browser Mobile Device Content Router API Gateway Catalog Service Review Service Order Service … Service Catalog Database Review Database Order Database … Database HTTP /HTML REST REST Browse & Search WebApp Product Detail WebApp ….
- 14. Database per service Private tables Private schema Private database server Catalog Service Review Service Order Service … Service Catalog Database Review Database Order Database … Database Note: NOT per service instance
- 15. @crichardson Loose coupling requires encapsulated data Order Service Customer Service Order Database Customer Database Order table Customer table orderTotal creditLimit
- 16. @crichardson How to maintain data consistency?!?!? Invariant: sum(open order.total) <= customer.creditLimit
- 17. @crichardson Cannot use ACID transactions BEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION Private to the Order Service Private to the Customer Service Distributed transactions
- 18. @crichardson 2PC is not an option Guarantees consistency BUT 2PC coordinator is a single point of failure Chatty: at least O(4n) messages, with retries O(n^2) Reduced throughput due to locks Not supported by many NoSQL databases (or message brokers) CAP theorem 2PC impacts availability ….
- 19. @crichardson How to do queries?
- 20. @crichardson Find recent, valuable customers SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ? …. is no longer easy Private to the Order Service Private to the Customer Service
- 21. @crichardson Microservices pattern language: data patterns http://microservices.io/
- 22. @crichardson Agenda Distributed data challenges in a microservice architecture Maintaining data consistency using sagas Implementing queries using API composition and CQRS
- 23. @crichardson From a 1987 paper
- 24. @crichardson Saga Using Sagas instead of 2PC Distributed transaction Order Customer Local transaction Order Local transaction Customer Local transaction Order Event Event X
- 25. @crichardson Event-driven, eventually consistent order processing Order Service Customer Service Order created Credit Reserved Credit Check Failed Create Order OR Customer creditLimit creditReservations ... Order state total … create() reserveCredit() approve()/reject()
- 26. @crichardson If only it were this easy…
- 27. @crichardson Rollback using compensating transactions ACID transactions can simply rollback BUT Developer must write application logic to “rollback” eventually consistent transactions Careful design required!
- 28. @crichardson Saga: Every Ti has a Ci T1 T2 … C1 C2 Compensating transactions T1 T2 C1 FAILS
- 29. @crichardson Order Service Create Order Saga - rollback Local transaction Order createOrder() Customer Service Local transaction Customer reserveCredit() Order Service Local transaction Order reject order() createOrder() FAIL Insufficient credit
- 30. @crichardson Sagas complicate API design Request initiates the saga. When to send back the response? Option #1: Send response when saga completes: + Response specifies the outcome - Reduced availability Option #2: Send response immediately after creating the saga (recommended): + Improved availability - Response does not specify the outcome. Client must poll or be notified
- 31. @crichardson Revised Create Order API createOrder() returns id of newly created order NOT fully validated getOrder(id) Called periodically by client to get outcome of validation
- 32. @crichardson Minimal impact on UI UI hides asynchronous API from the user Saga execution usually appears to be instantaneous (<= 100ms) If it takes longer UI displays “processing” popup Server can push notification to UI
- 33. @crichardson Sagas complicate the business logic Changes are committed by each step of the saga Other transactions see “inconsistent” data, e.g. Order.state = PENDING more complex logic Interaction between sagas and other operations e.g. what does it mean to cancel a PENDING Order? “Interrupt” the Create Order saga Wait for the Create Order saga to complete?
- 34. @crichardson Maintainability of saga logic Scattered amongst multiple services Services consuming each other’s events cyclic dependencies
- 35. @crichardson Order Service Orchestration-based saga coordination Local transaction Order state=PENDING createOrder() Customer Service Local transaction Customer reserveCredit() Order Service Local transaction Order state=APPROVED approve order() createOrder() CreateOrderSaga
- 36. @crichardson Reliable sagas require atomicity Service Database Message Broker update publish How to make atomic?
- 37. @crichardson 2PC still isn’t an option
- 38. @crichardson Option #1: Use database table as a message queue ACID transaction See BASE: An Acid Alternative, http://bit.ly/ebaybase DELETE Customer Service ORDER_ID CUSTOMER_ID TOTAL 99 CUSTOMER_CREDIT_RESERVATIONS table 101 1234 ID TYPE DATA DESTINATION MESSAGE table 84784 OrderCreated {…} … INSERT INSERT Message Publisher QUERY Message Broker Publish Local transaction ?
- 39. @crichardson Publishing messages Poll the MESSAGE table OR Tail the database transaction log
- 40. @crichardson Option #2: Event sourcing: event-centric persistence Service Event Store save events and publish Event table Entity type Event id Entity id Event data Order 902101 …OrderApproved Order 903101 …OrderShipped Event type Order 901101 …OrderCreated Every state change event
- 41. @crichardson Agenda Distributed data challenges in a microservice architecture Maintaining data consistency using sagas Implementing queries using API composition and CQRS
- 42. @crichardson Let’s imagine that you want to retrieve the status of an order…
- 43. @crichardson Order management is distributed across services Order Service Order status Delivery Service Delivery status Account Service Bill status
- 44. @crichardson A join is no longer an option SELECT * FROM ORDER o, DELIVERY d, BILL b WHERE o.id = ? AND o.id = d.order_id ABD o.id = b.order_id Private to each service
- 45. @crichardson API Composition pattern Order Service Order status Delivery Service Delivery status Account Service Bill status API Gateway GET /orders/id GET /bill?orderId=idGET /orders/id GET /deliveries?orderId=id
- 46. @crichardson API Composition works well in many situations BUT Some queries would require inefficient, in-memory joins of large datasets A service might only have PK-based API
- 47. @crichardson Find recent, valuable customers SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ?
- 48. @crichardson Using the API Composition pattern Order Service Order status orderTotal Customer Service Customer status creationDate API Gateway GET /orders/kind=recentHighValue GET /customers?creationDateSince=date GET /orders?status=SHIPPED &orderTotalGt=10000 Inefficient 😢
- 49. @crichardson Use Command Query Responsibility Segregation (CQRS)
- 50. @crichardson Command Query Responsibility Segregation (CQRS) Application logic Commands Queries X POST PUT DELETE GET
- 51. @crichardson Command Query Responsibility Segregation (CQRS) Command side Commands Aggregate Event Store Events Query side Queries (Materialized) View Events POST PUT DELETE GET
- 52. @crichardson Query side design Event Store Updater View Updater Service Events Reader HTTP GET Request View Query Service View Store e.g. MongoDB ElasticSearch Neo4J update query
- 53. @crichardson Persisting a customer and order history in MongoDB { "_id" : "0000014f9a45004b 0a00270000000000", "name" : "Fred", "creditLimit" : { "amount" : "2000" }, "orders" : { "0000014f9a450063 0a00270000000000" : { "state" : "APPROVED", "orderId" : "0000014f9a450063 0a00270000000000", "orderTotal" : { "amount" : "1234" } }, "0000014f9a450063 0a00270000000001" : { "state" : "REJECTED", "orderId" : "0000014f9a450063 0a00270000000001", "orderTotal" : { "amount" : "3000" } } } } Denormalized = efficient lookup Customer information Order information
- 54. @crichardson Queries database (type) Command side POST PUT DELETE Aggregate Event Store Events Query side GET /customers/id MongoDB Query side GET /orders?text=xyz ElasticSearch Query side GET … Neo4j
- 55. Benefits and drawbacks of CQRS Benefits Necessary in an event sourced architecture Separation of concerns = simpler command and query models Supports multiple denormalized views Improved scalability and performance Drawbacks Complexity Potential code duplication Replication lag/eventually consistent views
- 56. @crichardson Summary Microservices have private datastores in order to ensure loose coupling Transactions and querying is challenging Use sagas to maintain data consistency Implement queries using API composition when you can Use CQRS for more complex queries