Realizing the Event Driven Enterprise
- 1. Realizing the Event Driven Enterprise Enterprise Event Stream Reference Architecture Dave Reines, VP of Technology at Object Partners
- 2. Object Partners Building and Delivering Custom Software Solutions Since 1996
- 3. Object Partners Minneapolis Headquarters Omaha (60) Clients Nationwide 175 employees, all W-2 objectpartners.com
- 4. Practice Areas Platform Engineering, Real-time Data, Modern APIs, Mobile/Web, UX
- 7. Point-to-Point Integrations • Project specific integrations • Limited re-use • Tightly coupled systems • Difficult to maintain • Increased vendor lock-in • Batch-oriented processes
- 8. Architectural Objectives To provide near real-time, standardized, synchronization of mission critical data originating from various heterogeneous data sources in a secure, scalable and highly available manner
- 10. Enterprise Event Stream • Provides single, standardized • Accommodates real-time and batch integrations • Enables modern platforms • Public Cloud Enablement • Real-time analytics • Microservices • Alternative datastores (Opensource, Cloud, NoSQL) • Notifications and alerting • Offloads legacy systems
- 14. Legacy Sources • DB2, Oracle, SQL Server, MySQL, etc. • Change data capture (CDC) publishes inserts/updates/deletes in near real-time • Typical CDC mechanisms minimize impact on legacy sources by utilizing DB transaction logs
- 15. Raw Topics
- 16. Raw Topics • Raw topic for each database table • Message published per insert/update/delete • Each message contains a full copy of the record • Topics are eventually compacted to a single message per primary key (reducing storage and full-load time)
- 18. Source Aggregator • Consumes each raw topic for a given source • Builds (domain-driven design) from associated entities • Uses aggregation state store to aggregate entities • Publishes aggregate entities to aggregate topics
- 19. Aggregate Topics
- 20. Aggregate Topics • Real-time enterprise • Message published per insert/update/delete to any part of the aggregate • Each message contains a full copy of all aggregate data • Topics compacted to a single message per aggregate key (reducing storage and full-load time) • Deltas to support “business events”
- 22. Materialized Views • Entities consumed from aggregate topics and stored in one or more materialized views • Materialized views accessed via APIs • Datastore technologies selected based on required query patterns (i.e. key lookup, search, etc.) • Views bootstrapped and versioned via compacted aggregate topics
- 23. APIs
- 24. Data Storage
- 25. • Each record stored four or more times • Source system • Raw Topic (Compacted) • Aggregation State Store • Aggregate Topic (Compacted) • Materialized View (one or more) • API Versioning Data Storage
- 26. • Raw Topic Compaction • Decreases full load time • Aggregate Topic Compaction • Provides fast bootstrapping of new materialized views • Provides fast re-indexing of materialized views • Eliminates impact on other consumers during bootstrapping and re-indexing • Tiered Storage • KIP-405: Kafka Tiered Storage Compaction Benefits
- 28. • Raw Topics and/or Aggregate Topics setup with short ttl (i.e. seven days) • Backup and replay via less expensive object storage (i.e. s3) • Reduces topic storage requirements • Potentially reduces storage costs Short Time-to-live • Increases complexity • Engineering • Disaster recovery • Replay path rarely executed • Increased operational risk • All consumers impacted on full-load
- 30. Dual Writes
- 31. • Redundant active/active environment • Dual writes from source • Maximum Availability • Mirroring may reduce availability in DC outage • Active/Standby availability during full loads • Blue/Green deployments • Full loads • Versioning Dual Writes
- 33. • Aggregation processing in a single data center • Replicate aggregate topics • Materialized views hydrated via aggregate topics in each data center • Eliminates replicated raw topics and state store • Active/Active API availability • Active/Standby aggregation • Requires raw topic full load on DC failover Aggregate Topic Replication
- 34. Aggregation in the Cloud
- 36. Materialized View Replication ● Generally ● Datastore specific replication strategy ● Limited event triggering ● Replay in a single data center replicates all data
- 38. Migration Patterns Incremental vs Big Bang
- 41. Architectural Objectives To provide near real-time, standardized, synchronization of mission critical data originating from various heterogeneous data sources in a secure, scalable and highly available manner