Microservices event-driven-saga
- 1. Microservices BankNext’s -Event Driven- SAGA Compensation
- 2. Business Objective • “BankNext” employed event driven choreography & achieved high processing efficiencies as described in architecture-as-is • However, when reconciling customer & account information, abnormal system behavior was noticed • Problem scenarios - • Dangling Customers - Active Customers without any valid Account • Linkage not created - Valid Customers and Accounts present but linkage between these entities is missing • How did this happen ? Let’s investigate
- 3. Current Event-Driven w/o SAGA
- 4. Back to the Drawing Board • BankNext’s event driven system is composed of multiple msvcs that independently manage data persistence • If any of the msvcs in this transactional flow fail, then there is no provision to correct/rollback the data that was already persisted by an upstream msvc • This was the root cause of severe data integrity problems in the system
- 5. Technical Challenge • The “CustomerMgt” msvc successfully creates the customer entity in the “customer” table in the RDBMS • Next, “AccountMgt” msvc creates the account entity in the “account” DB table , after receiving the subscription on the Kafka topic • In the last step, “EntityAggregation” msvc links these 2 entities by creating an entry in to the “customer_account_map” table in the RDBMS • The process is considered successful if & only if the above 3 steps are atomic i.e. Either all 3 steps complete successfully or none complete • If/when the “AccountMgt” or “EntityAggregation” msvc fails for any reason then the data integrity problems occur
- 6. Scenarios 1. Example scenario i. Customer entity created successfully. ii. Account Msvc fails the request per the business validation rule iii. eg. only “SAVINGS” & “CURRENT” allowed but “OVERDRAFTS” received iv. Account entity creation aborts. v. Consequence - Customer is left without an Account. Thus Dangling Customer. 2. Example scenario i. Customer entity created successfully. ii. Account entity created successfully. iii. EntityAggregation Msvc fails the request per the business validation rule iv. eg. for “USA” only type allowed is “SAVINGS” but “CURRENT” received v. Customer and Account linking aborts. vi. Consequence - Active Customer and Account is created but they are not linked
- 7. Solution Approach SAGA compensation 1. In such a failure scenario, the solution requires that all the data persisted by upstream msvcs be rolled back 2. Thus the system data can be brought back to the state that it was before this txn started 3. SAGA approach is recommended to accomplish this task
- 9. Technical Implementation (contd..) SAGA compensation 1. New Kafka topics are created for capturing messages in case of failure scenarios - i. customer_failure_topic ii. account_failure_topic iii. entitymap_failure_topic 2. When a msvc fails, a message is published to the corresponding failure topic 3. When “AccountMgt” publishes to the “account_failure_topic”, i. “CustomerMgt” which is subscribed to this topic, gets notified ii. DB rollback activity initiated. 4. When “EntityAggregation” publishes to the “entitymap_failure_topic”, i. “AccountMgt” & “CustomerMgt” which is subscribed to this topic gets notified ii. DB rollback activity initiated 5. Thus SAGA compensation accomplishes txn atomicity and restores the original system state
- 10. SAGA Interactions – Closer Look
- 11. State of the Database SAGA-Customer-Rollback
- 12. State of the Database SAGA-Account-Rollback
- 13. State of the Database Customer-Account-Integration-Success
- 14. New Architecture Positives 1. SAGA enables robustness and graceful system integrity restoration 2. Provides a sturdy & flexible mechanism to accomplish process atomicity 3. Eliminates tight coupling via publish & subscribe, as needed
- 15. New Architecture Negatives 1. SAGA adds to system complexity and maintenance 2. Observability, system debugging and tracing capabilities need to be significantly ramped up 3. There is a danger that the rollback operation triggered by the SAGA may itself fail. 4. Retries may be needed in such scenarios which further complicates matters
- 16. Summary : Architecture, TechStack & Rationale Github : https://github.com/vijayredkar/event-driven-platform-with-saga Architecture Goal Strategy Techstack Rationale Flexible & extensible integrations Choreography Topics Eliminates centralized coordinator dependency Event driven capability Messaging systems Kafka Eliminates tight coupling btwn svcs Transactional system data integrity SAGA Topics - failure capture Self healing system recovery Future capability efficiencies Publish/Subscribe Kafka Eliminates explicit new svc call invocation System wide observability Centralized logging/tracing ELK/Sleuth Async logging & observability Heavy volume/TPS Streaming Kafka/Async operations Extremely low latency Parallel Aggregation CompletableFuture Java General purpose Framework microservices Springboot General purpose Transactional Durability Persistence RDBMS (H2 for demo) General purpose
- 17. Thank you! -Vijay Redkar Github: https://github.com/vijayredkar Medium: https://vijayredkar.medium.com/ vijay0512@yahoo.com +971 588823757