Design-Time vs. Runtime: Understanding Topics and Events in Event-Driven Architectures

In my previous article, Understanding Topics and Events: The Foundation of Modern EDA, I shared my thoughts on the foundational concepts of topics and events within Event-Driven Architecture (EDA), aiming to equip architects, developers, and business stakeholders with insights into how these elements drive successful EDA implementations.

Building on that foundation, this article extends the discussion to explore topics and events from both design-time and runtime perspectives. Understanding these perspectives is crucial for creating robust and scalable EDA systems.

From a runtime perspective, this understanding ensures system performance, real-time data flow, and alignment between design and implementation, enabling a seamless and effective EDA strategy.

1. Design-time Perspective

At design-time, aligning events and topics wit business logic and establishing clear communication contracts prevents ambiguities that might cause runtime issues. Well-structured topics support scalability and efficient routing, while thoughtfully defined events encapsulate precise business actions.

Events: Defining What Happens

Purpose:

The goal at design-time is to architect a system where events and their interactions are well-structured, predictable, and aligned with business logic. The design phase defines how components communicate, what information is shared, and how it is organized for efficient processing.

Focus:

Revolves around data modeling and domain understanding.

• Schema Definition: Define the structure (e.g., JSON Schema, Avro) and content of events. For example, an OrderPlaced event might include fields like orderId, userId, totalAmount.
• Event Naming: Use clear, descriptive names reflecting the domain (e.g., UserSignedUp or PaymentProcessed).
• Granularity: Decide how specific or generic events should be (e.g., separate events for OrderCreated and OrderConfirmed or a single OrderUpdated event).
• Versioning: Plan for schema evolution to maintain compatibility as the system grows.
• Producer-Consumer Relationships: Designing how event producers, message brokers, and consumers interact.

Key questions:

• What real-world occurrences need to be captured?
• What data should the event payload contain?
• How should events be named to reflect intent (e.g., OrderPlaced, PaymentFailed)?

Concerns:

• Event schema: Define attributes, data types, and versioning strategies
• Event granularity: Ensure events are neither too generic nor overly specific.
• Schema Evolution: Managing backward and forward compatibility when event structures change.
• Redundant or Confusing Event Definitions: Inconsistent naming or unclear purposes for events.

Topics: Structuring Communication

Purpose:

The design-time focus is on defining the structure and organization of topics to ensure clarity, scalability, and logical data flow. Proper topic planning reduces complexity and improves system communication efficiency.

Focus:

Revolves around system communication patterns and event organization.

• Topic Hierarchy: Design a structure that reflects the domain and makes it easy to discover relevant topics. For example, use ecommerce/orders/created for an order creation event.
• Domain Alignment: Ensure topics align with business contexts to avoid cross-domain dependencies.
• Access Control: Design topic-level permissions to secure sensitive data.
• Wildcards in Hierarchical Topics: Routing flexibility, and enable fine-grained access control.

Key Questions:

• What are the primary data categories or business domains that topics will represent?
• How should topics be structured (e.g., flat vs. hierarchical naming) for scalability?
• How should topics align with business domains for maintainability?
• How can we minimize cross-domain dependencies?

Concerns:

• Overly Complex Topic Structures: Excessive granularity can lead to unmanageable systems.
• Broad, Unclear Topics: Grouping unrelated data can reduce processing efficiency.
• Lack of Consistency in Naming: Inconsistent names create confusion and maintenance issues.
• Topic Explosion: Unplanned growth in topics can overwhelm infrastructure.

If you are curious about hierarchical topics, checkout Solace Topic hierarchy design and best practices.

2. Runtime Perspective

Events: Data in Motion

Purpose:

At runtime, the system executes the event-driven architecture by emitting, processing, and reacting to events. The focus shifts from planning to real-time handling of data flows, failures, and performance management.

Focus:

• Immutability: Events are immutable records of what happened, ensuring traceability.
• Serialization: Events are serialized for transmission (e.g., JSON, Avro, Protobuf).
• Delivery Guarantees: Events are delivered based on agreed guarantees:

"At most once" Fast but no retries if delivery fails.

"At least-once" Ensures delivery but may result in duplicates.

"Exactly once" Guarantees a single delivery but at a higher cost.

• Monitoring and Reliability: Ensuring system stability and observing metrics.

Key Questions:

• How are events published, subscribed to, and consumed?
• What mechanisms handle retries, failures, and dead-letter queues?
• How is event flow monitored for performance bottlenecks?
• What patterns and features available to handle event ordering?

Concerns:

• Data integrity and delivery guarantees.
• Serialization format: JSON, Avro, or Protobuf.
• Delivery guarantees: At-most-once, at-least-once, or exactly-once.
• Event Delivery Failures: Network issues or consumer downtime may cause message loss.
• Processing Delays: High throughput systems may face bottlenecks.
• Idempotency and Duplication: Without proper handling, duplicate events can lead to inconsistent state.

Topics: Routing in Action

Purpose:

At runtime, the system dynamically uses topics for routing, publishing, and consuming messages. The focus is on ensuring that the topic design supports efficient real-time processing, error handling, and scaling.

Focus:

• Message Publishing and Subscribing: Producers send events to topics, and consumers subscribe to them.
• Dynamic Subscriptions: Consumers subscribe to specific topics of interest to receive relevant events.
• Routing and Filtering: Using topics to manage event flow to the right consumers and reduce unnecessary data transfer.
• Monitoring Topic Utilization: Tracking message volume and performance.

Key Question:

• How are messages published and consumed using topics?
• How to ensure optimal parallel processing?
• What routing and filtering logic is needed for message flow?
• How is topic usage monitored to detect performance issues?

Concerns:

• Topic Overload: Too many consumers on a topic may reduce performance
• Message Routing Failures: Incorrect topic routing can lead to data loss.
• Insufficient Monitoring: Lack of observability hinders troubleshooting.
• Dynamic subscription management: Consumers subscribe to relevant topics

Ok, let us summarize.

Conclusion

By differentiating the roles of events and topics across design-time and runtime, you can create clear, maintainable architectures while ensuring efficient and reliable system performance. This perspective ensures teams design with foresight and execute with precision.