Kafka Summit 2022: Handling Eventual Consistency in a Transactional World.pdf
0 likes19 views
The document discusses handling consistency challenges when using a digital integration hub (DIH) with an event-driven architecture and Kafka as the event streaming platform. It describes three patterns for enforcing consistency: (1) using an outbox pattern at the source system, (2) using a callback pattern without modifying the source, and (3) buffering events in Kafka until transactions are closed. It also presents a cross-docking pattern that buffers events in a fast storage system before writing business events. Maintaining consistency in distributed systems is challenging, and the presenters evaluate different approaches and their tradeoffs.
Kafka Summit 2022: Handling Eventual Consistency in a Transactional World.pdf
- 1. KAFKA Summit EMEA 2022 Andrea Gioia CTO at Quantyca Co-Founder at Blindata Matteo Cimini Data Architect at Quantyca Handling Eventual Consistency in a Transactional World
- 2. Who we are? Not an easy question to answer but keeping it simple... Andrea Gioia CTO & Partner andrea.gioia@quantyca.it Matteo Cimini Data Architect Quantyca is a privately owned technological consulting firm specialized in data and metadata management based in Italy quantyca.it matteo.cimini@quantyca.it
- 3. Digital Integration Hub Where we left off System of engagement System Of Insight System of Records Legacy Systems Application Layer Digital Integration Hub API Gateway Event-Based Integration Layer High-Performance Data Store Microservices Metadata Management Data offloaded from legacy systems are aggregated in read models stored into dedicated low-latency, high performance datastores accessible via APIs, events or batch. The data store synchronizes with the back ends via event-driven integration patterns. Benefits ○ Responsive user experience ○ Offload legacy systems from expensive workloads generated by front-end services ○ Support legacy refactoring ○ Align services to business domain ○ Enable real time analytics ○ Foster a data centric approach to integration
- 4. PROS + Can handle very high throughput CONS - Not a good fit for complex events processing - TCO may not be optimal for huge data volumes PROS + Low Latency + Can handle very high throughput + Simplified schema management CONS - Not a good fit for complex stateful transformations - Can have some performance issues at very high throughput PROS + Largely used by service developers, probably already present in the architecture + Simplified schema management CONS - Not a good fit for complex events processing - Can have some performance issues at very high throughput PROS + SQL Compliant + Transactional (ACID) + TCO can be optimized selecting the right storage strategy between RAM and disk CONS - Can have some performance issues at very high throughput - Low latency is not guaranteed Event Store The kafka option RDBMS NoSQL DB Streaming Platform Distributed Cache
- 5. Event store on Kafka Main consistency challenges TRANSACTIONAL CONSISTENCY: The read model must be consistent from a transactional point of view with the upstream source aggregate offloaded form source system. Kafka is not a transactional system ORDERING CONSISTENCY: The read model must be updated only in forward way. Older states cannot replace newer ones. For complex source aggregates is very easy to have events delivered out of order. HISTORICAL CONSISTENCY: The read model must be easily created at anytime from scratch without information loss. Infinite retention in kafka is possible but it is not always a faisable option. Event Store Target Read Model Source Aggregate
- 6. Event store on Kafka Source System Event Store Technical Events (Speed & Fidelity) Domain Events (Trusted Views) High Performance Data Store Business Events (Ease of consumption) Commands Micro/Mini Services READ WRITE Handling consistency challenges in a DIH Aggregate Read Model
- 7. Handling consistency challenges at the source Outbox pattern Source System Streaming Platform Technical Events (Speed & Fidelity) Domain Events (Trusted Views) DESCRIPTION The source system is modified in order to inserts messages/events into an outbox table as part of the local transaction. The modification can be performed at code or database level (es. triggers or materialized views). The connector that offloads data to the streaming platform is triggered by the outbox table. PROS + It has no overhead in terms of latency and throughput + It does not generate extra workloads at the source CONS - It’s not always possible to modify the source to implement this pattern OUTBOX Table COMMIT TRX INSERT UPDATE DELETE INSERT
- 8. Handling consistency challenges at the source Callback Pattern Source System Streaming Platform Technical Events (Speed & Fidelity) Domain Events (Trusted Views) DESCRIPTION All changes to tables that are part of the same aggregate are mapped to the same topic as technical event that can contain only the aggregate id and transaction id as payload. For every transaction id a stream processor query the legacy database extracting the modified aggregate, filtering by id, and publishing it as payload of a new domain event To reduce the workload on legacy the stream processor can query a read replica PROS + Do not require any modification at the source CONS - Even if the footprint at the source is less then a standard pooling solutions it could be anyway not trivial especially for high throughput transactional sources
- 9. Handling consistency challenges in kafka In-flight handling pattern Domain Events (Trusted Views) Kafka Streams Ecosystem Buffering Events Closing Transactions Ordering Transactions Transactions Metadata DESCRIPTION All events that are part of the same aggregate are buffered to the same changelog topic until the corresponding END transaction event has been captured. Closed transactions are then ordered by a Punctuation function and mapped to the corresponding domain event. PROS + It has minimum overhead in terms of latency and throughput + It does not generate extra workloads at the source + Do not require any modification at the source CONS - Need to write stateful applications - Need to use Processor API capabilities (low level statestore, punctuators, etc etc) - Transaction Metadata must contain END transaction informations Streaming Platform Technical Events (Speed & Fidelity)
- 10. Transactions Metadata Buffering Layer Handling consistency challenges in the fast storage Cross-docking pattern Technical Events (Speed & Fidelity) Domain Events (Trusted Views) Fast Storage Closing & Ordering Transactions Data Model Business Events (Ease of consumption) Streaming Platform DESCRIPTION Events that are part of the same aggregate are buffered in the same buffering table until the corresponding END transaction event has been captured. Closed transactions are then ordered by a micro-batch processor and then mapped to the corresponding domain event. PROS + It has medium overhead in terms of latency and throughput + Stateful solution on a stateful storage (typically strong consistent) + Fast Storage is SQL compliant in most cases + Business Events can be enriched with external informations CONS - Need to add a component (Fast Storage) to the overall architecture (DIH)
- 11. Takeaways In a complex and heterogeneous architecture not all consumers can handle eventual consistency There are different solutions to enforce consistency in an event driven architecture like the digital integration hub There are no free lunches anyway. Every solution comes with pros and cons. It is important to evaluate them in the context. The general rule of thumbs is to ○ use the outbox pattern based solution for every newly implemented custom source (and for all legacy sources you are allowed to modify) ○ decide between in-flight handling and cross docking pattern based solution for existing solution considering latency trade off and the skills set of your engineering team What’s next… We are working to define solution templates at the platform level in order to provide to data product teams consistency preserving services in a self service way through a declarative interface. More on this next year ;-)
- 12. Questions? Feel free to ask matteo.cimini@quantyca.it andrea.gioia@quantyca.it
- 13. Corso Milano, 45 / 20900 Monza (MB) T. +39 039 9000 210 / F. +39 039 9000 211 / @ info@quantyca.it www.quantyca.it