A Deep Dive into Event Sourcing: Immutable and Scalable Systems
- 1. 1 A Deep Dive into Event Sourcing Immutable and Scalable Systems Presenter:
- 2. 2 Table of Contents • Traditional State-Based systems • Optimistic Concurrency • Limitations • Introducing the Time Dimension • Streams & Events • Event Stores • State Reconstruction • Aggregates • Projections • Commands • Queries • CQRS • Event Sourcing Compared to Other Event-Driven Terms • Challenges • Conclusion • Further Readings
- 3. 3 Traditional State-Based systems • Traditional state-based systems typically store data as records in relational tables, documents in collection-based databases, or similar formats. • The key characteristic shared by these systems is that they store only the most recent or final state of each record, overwriting previous states.
- 4. 4 Optimistic Concurrency • Initial State: Both Maysam and Jihad load the same order record, each with a copy that includes a version column indicating the current version. • Concurrent Updates: Each of Maysam and Jihad issues an update for the same record. • Optimistic concurrency uses: • Version Increment: When someone updates the record, the version column in the database increments. • Predicate Filtering: The update command includes a where condition, ensuring it only applies if the version in the database matches the version in memory. • Outcomes: • Maysam’s update succeeds since it reached the database first, incrementing the version. • When Jihad’s update arrives, the version mismatch triggers a concurrency exception, as his update is based on an outdated version.
- 5. 5 Limitations • No Historical Context: Only the latest data is saved, so changes leading to the current state aren’t preserved. • Limited Auditability: Without built-in tracking, auditing or tracing changes is complex and often unreliable. • Data Recovery Challenges: Reverting to previous states is difficult, as historical snapshots aren’t available. • Fewer Insights: Lacking event history, these systems miss patterns and trends useful for decision- making. • Dependence on External Tracking: Additional systems are often needed to record history, increasing complexity. • Harder Debugging: Without event logs, identifying the root cause of issues can be challenging.
- 6. 6 Introducing the Time Dimension • We were exploring the limitations of traditional state-based systems—so, are there any solutions? • Can we track every single change made to entities or records? • What if we made the concept of time a first-class citizen, giving it top priority in how we handle data? • Meaning • Comprehensive logs and audit trails would be available for deeper analysis, insights, and more. • Questions like "What happened?" and "Why did it happen?" would be much easier to answer. • Debugging and troubleshooting would become significantly simpler. • Etc … • That’s exactly what event sourcing is about! Where state stored as a series of events.
- 7. 7 Streams & Events • Stream: A collection of related events that occurred over time, often linked to a particular entity/aggregate. • Event: A record of something happened in the system. • Example: The order with ID 321-31 is stored as a sequence of events. Every event in the system is recorded, providing a comprehensive audit log. • Key Benefits: • Enhanced Debugging: Easier to trace and understand issues. • State Reconstruction: Rebuild the state at any point in time. • Data-Driven Insights: Enables customer behavior analysis, machine learning models, and more, empowering the business to make impactful decisions.
- 8. 8 Migration
- 9. 9 Event Stores • Is a type of database designed specifically for saving events instead of just the current state of data. • Characteristics: • Streams as primary building blocks. • Immutable Events: Once recorded, events can’t be modified. • Append-Only: Events are added without altering data. • Time-ordered events: Events are stored in a chronological order. • Optimized for event retrieval. • Concurrency handling. • Subscriptions: Support for notifying other systems of new events. • Off the shelf solutions: • EventStoreDB • Is it possible to build customized event store using a relational or document DBs? • Yes, it’s possible to use a relational database or certain NoSQL databases like MongoDB. • The key lies in understanding the principles of an event store and implementing the necessary characteristics to meet the use case requirement.
- 10. 10 State Reconstruction • While events capture valuable data, they don't inherently provide immediate value. Relying on raw events alone presents challenges in meeting various system requirements. • Read Concerns: • UI & Data Presentation: Frontend and UI components require structured data to display information clearly to users. Events alone lack this structure, making it difficult to support straightforward data views. • Reporting Complexity: Traditional reporting aggregates data in structured formats but achieving this with events is challenging since events may not follow a standardized schema. • Write Concerns: • Lack of Immediate State: In traditional systems, business logic relies on the current state of a record. With event sourcing, there's no single, accessible "current state"—only a series of events representing changes over time. • Solution: Rehydration, we must obtain the entity’s current state by rehydrating the events in order. • Aggregates and Projections are used as containers for the reconstructed state. • Aggregates handle business logic.
- 11. 11 Aggregates • It’s a Domain-driven design tactical pattern. • It’s a cluster of associated domain objects that we treat as single unit for the purpose of data change. • The aggregate must enforce business rules and domain invariants. It represents a transactional boundary. • In the context of event sourcing • An aggregate replays events to reconstruct its current state. Each event represents a meaningful change that happened in the past. • When an aggregate is rehydrated, it applies these events in sequence to build its latest state.
- 12. 12 Projections • Specialized views. • Since events capture changes over time but don't directly represent the current state of an entity, projections are created by processing and transforming these events into a format that’s optimized for reading. • Essentially, a projection "projects" the historical event data into a present, readable state. • Projections let you leverage historical event data to create real-time views tailored to various application needs, without needing to change the underlying event structure.
- 13. 13 What is the difference? • Since both aggregates and projections serve as containers for holding the current state of an entity, what distinguishes them? • The difference lies in their purpose: • Aggregates represent the reconstructed state and enforce domain rules when handling state-changing actions or commands. • Projections represent the reconstructed state specifically for read-only actions or queries. • Pay attention that in both use cases, our source of truth is events themselves, we are reconstructing state by replaying events in order.
- 14. 14 Order Aggregate • In the context of event sourcing, generally, an aggregate will be composed of • Properties to represent its current state • Behaviours or functions that modify the aggregate’s state • Helper functions for reconstruction, which replay events and update the aggregate's fields based on each event type • Remember, aggregates are designed for state changing/commands.
- 15. 15 Commands • A command is simply an instruction that changes the state of our system. • In event sourcing, • since event stores are append-only, every command is expected to produce an event as its output. • Typical flow of command execution 1. Fetch aggregate/entity’s events 2. Rehydrate the current state of the entity by replaying events in order 3. Perform the command 4. Commit the produced command to the event store
- 17. 17 Optimistic Concurrency – Event Sourcing • In this scenario, optimistic concurrency control is working as intended to prevent conflicting updates on the same order. • Here’s a breakdown of the process: • Loading Events: Both Maysam and Jihad load the same set of events corresponding to an order, capturing the current version of the data. • Command Execution: Each performs their own command, resulting in a new event based on the same initial version of the order. • Commit to Event Store: • Maysam’s Commit: Maysam commits first, so his event gets stored successfully. This commit increments the version of the order in the event store. • Jihad’s Commit: When Jihad tries to commit his event, the event store detects that the version he loaded is now outdated (since Maysam’s commit has incremented the version). As a result, the event store throws a concurrency exception for Jihad. • Only the first commit based on the current version is accepted; later commits face a concurrency exception. • Jihad must reload the latest version, including Maysam’s update, and reapply his command.
- 18. 18 Order Summary Projection • In the context of event sourcing, generally, a projection will be composed of • Properties to represent its current state • Helper functions for reconstruction, which replay events and update the projection's fields based on each event type in a customizable way to fulfil the need of the use case. • Remember, projections are designed for queries. • They lack behaviours as well, because their primary goal is handling queries.
- 19. 19 Queries • Requests for information that retrieve and display data without changing it. • In event sourcing • There is nothing special, queries are expected to return data, so no events will be produced or committed, just data retrieval. • Typical flow of query execution 1. Fetch aggregate/entity’s events 2. Project the current state of the entity by replaying events in order in a customizable way to fulfil the need of the use case. 3. Return the projected result
- 21. 21 CQRS • The flow diagram on the right summarizes the basic process we followed for commands and queries • Aggregates manage commands. • Projections handle queries. • This approach is known as Logical CQRS, as it separates read and write responsibilities into distinct models, each with its own role and purpose. • Currently, our event store serves as the primary source of truth for both commands and queries. • Moving Towards Higher CQRS Separation • To enhance performance and scalability, we could set up a dedicated read database specifically for handling queries. • This would allow queries to be processed directly from an optimized data source rather than from the event store itself.
- 22. 22 CQRS (Cont.) • Our event store serves as the source of truth, providing the foundation for rebuilding the current state when processing commands. • For reads, we rely on a separate, specialized read database, optimized to meet specific use cases. • Synchronization • The event store and read database are kept in sync asynchronously, achieving eventual consistency. • Typically, the EventStore supports a subscription mechanism, allowing event handlers to subscribe to new events as they occur. • This setup is often called Physical CQRS, and it is the most common. • Advantages of Physical CQRS: • Improved Performance for Reads and Writes. • Optimized Data Models. • Scalability. • Better Support for Eventual Consistency. • Enhanced Reliability and Availability. • Flexible Technology Choices. • Easier Reporting and Analytics. • Question: is logical CQRS still applicable in this context?
- 23. 23 Event Sourcing VS Change Data Capture • Definition: A design pattern where changes to application state are stored as a sequence of immutable events rather than overwriting the current state. • Richer Context: ES can provide richer context about changes of “Why” and “How” they happened because they capture events meaningful to the business. • We can see • Why: The deposit happened because the customer initiated it. The bonus happened as part of a loyalty program. • How: Through explicit events capturing user actions or automated rewards. • Definition: A technique to capture and track changes in a database in real time and propagate them to other systems. • Limited Context: CDC tracks data changes (e.g, UPDATE, DELETE) but lacks the business intent behind those changes. • As we can observe • Captures only the database changes: Balance: 3200 -> 3250 - > 3260. • Why/How is unclear: It doesn't indicate that a deposit or bonus caused the change, nor their reasons.
- 24. 24 Event Sourcing VS Event Streaming VS Event-Drive Architecture • Event Sourcing: • Definition: A pattern where the state of an entity is derived from a sequence of immutable events stored in an event store. Each event represents a change in the entity’s state. • Example: Tracking the history of order processing by storing events like “OrderCreated” and “OrderPlaced”. The current state of the order is determined by replaying these events in sequence. • Event Streaming: • Definition: A real-time, continuous flow of events (data) sent over a messaging system (e.g., Kafka) for processing and analysis. • Example: Database changes captured using CDC tools are streamed through Kafka into a reporting database, enabling real-time analytics and insights. • Event-Driven Architecture: • Definition: A design paradigm where services communicate by producing and consuming events, reacting asynchronously to changes or triggers in the system. • Example: When a customer places an order, an “OrderPlaced” event triggers independent services for payment processing, inventory updates, and shipment tracking, each reacting to the event asynchronously.
- 25. 25 ES vs ES vs EDA (Cont.)
- 26. 26 Challenges • Event Versioning. • Storage Growth. • Complexity. • Consistency. • Rehydration Performance. • Steep Learning Curve. • And Others.
- 27. 27 Conclusion • Event sourcing is about capturing the current state as a series of events. • A stream refers to a collection of events associated with a specific entity. • Event stores are specialized databases used to persist these events. • Since events alone don't provide value, reconstructing the state from these events is necessary to realize their benefits. • Aggregates serve as structures to represent the current state of entities for handling commands, while projections represent the state for queries. • The separation of models for commands and queries is known as logical CQRS. • A more advanced level of CQRS can be achieved by using a dedicated read database for query handling, often referred as physical CQRS. • Event sourcing introduces certain challenges, such as increased complexity, handling versioning, and ensuring consistency, etc.. which require substantial practice and experience to manage effectively. • Event Sourcing provides richer context compared to Change Data Capture (CDC). • In event sourcing, events are the source of truth, ensuring an immutable log of all state changes. In contrast, CDC primarily captures changes at the database level, which may eventually be deleted or overwritten, losing historical context. • Event Streaming refers to the continuous real-time flow of events, enabling systems to process data as it is produced. • Event-Driven Architecture (EDA), on the other hand, focuses on asynchronous communication between system components, where events act as triggers to facilitate decoupled and reactive interactions.
- 28. 28 Further Readings • Beginner's Guide to Event Sourcing | Event Store • Snapshots in Event Sourcing for Rehydrating Aggregates | CodeOpinion • Why is Kafka not Ideal for Event Sourcing? | Domenic Cassisi • Marten | .NET Transactional Document DB and Event Store on PostgreSQL • Let's build event store in one hour! | Oskar Dudycz • MiniTrello (Plain Event Sourcing – No Libraries) | GitHub • Distributed Data for Microservices - Event Sourcing vs. Change Data Capture | debezium