Event sourcing Live 2021: Streaming App Changes to Event Store
0 likes116 views
This document discusses streaming app changes to event stores. It covers change data capture (CDC) which involves identifying, capturing, and delivering changes made to data. CDC can be done by capturing events from app code or by tailing database transaction logs. Capturing from app code provides flexibility but requires extra code, while capturing from databases is easier to control but depends on database log formats. The document also discusses using event stores for data warehouses, data lakes, CQRS patterns, and hybrid transactional/analytical processing (HTAP) databases.
Event sourcing Live 2021: Streaming App Changes to Event Store
- 1. Streaming App Changes to Event Store Shivji Kumar Jha SME, Stream Platform, Nutanix
- 2. About Me Platform Engineer Specialties: DBs, Stream, SOA, infra etc Love Distributed systems Open source contributions Apache Pulsar, MySQL Communities & conferences! Talks https://www.slideshare.net/shiv4289/ https://www.youtube.com/watch?v=Bx4csRi1b8Y&list=PL A7KYGkuAD071myyg4X5ShsDHsOaIpHOq @ShivjiJha in/shivjijha
- 3. The days of Monolith One application Data in one place Easy to query and fetch information
- 4. Microservices are hard! Move ALL transactional data to one store – event store Can’t query all data at once 1.Aggregate 1.Load 1.Index 1.Query Data is fragmented. Application split into modular microservices
- 5. Event Store – use cases Data warehouses Data Lakes Data Lakehouse CQRS
- 7. Change Data Capture (CDC) How Send Events from App Code Tail database
- 8. CDC: from App Code Capture Bad Write code for each event Difficult to control from platform teams.
- 9. CDC: from App Code Capture Bad Write code for each event Difficult to control from platform teams. Good Can leverage Flexibility in code App has more data than database. Eg : Data aggregated from other apps
- 10. CDC: from App Code Capture Bad Write code for each event Difficult to control from platform teams. Good Can leverage Flexibility in code App has more data than database. Eg : Data aggregated from other apps Delivery: Mostly asynchronously Synchronous if critical transaction, extra latency is ok.
- 11. CDC: from database Capture: Tail transaction logs
- 12. CDC: from database Capture: Tail transaction logs Bad Different structure in transactional logs of each DB Deal with log format changes in upgrades. Deal with DDL changes Maybe DB reverted transaction while recovering..
- 13. CDC: from database Capture: Tail transaction logs Bad Different structure in transactional logs of each DB Deal with log format changes in upgrades. Deal with DDL changes Maybe DB reverted transaction while recovering.. Good Easy control from platform team. No extra code. No extra latency. App and CDC decoupled via queue and offset. Standard tools available : Debezium, maxwell
- 15. Hybrid Databases (HTAP) HTAP: Hybrid Transactional/Analytical Processing Evolving Landscape One database to serve OLTP and OLAP Examples: SingleStore DB TiDB MySQL Heatwave
- 16. References Change Data Capture https://en.wikipedia.org/wiki/Change_data_capture Debezium https://github.com/debezium Maxwell http://maxwells-daemon.io/ https://github.com/zendesk/maxwell https://medium.com/@purbon/patterns-to-track-changes-in-data- 8d239734dc32 Outbox pattern https://medium.com/engineering-varo/event-driven-architecture- and-the-outbox-pattern-569e6fba7216