fundamentalsofeventdrivenmicroservices11728489736099.pdf
0 likes15 views
The document provides a comprehensive overview of event-driven microservices, focusing on communication structures, service modeling, and event modeling. It discusses the challenges of modern architectures, including data management and scaling with synchronous services, while emphasizing the importance of event streams as a single source of truth. The content is structured in three parts: foundational concepts, event modeling techniques, and practical service modeling examples to guide effective implementation of microservices.
![(Almost) self-documenting
{
"type" : "record",
"name" : "User",
"namespace" : "com.example",
"fields" : [
{"name": "username", "type": “string"},
{"name": "phone", "type": [null,"int"], "default" : null},
{"name": "address", "type": "string", "default": “UNKNOWN"},
{"name": "country", "type": "enum", "symbols": ["CA", "US", "OTHER"],
"doc" : "The user's country. Select other if not in list"}
]
}](https://image.slidesharecdn.com/fundamentalsofeventdrivenmicroservices11728489736099-241021085110-2c61dcf9/85/fundamentalsofeventdrivenmicroservices11728489736099-pdf-68-320.jpg)
fundamentalsofeventdrivenmicroservices11728489736099.pdf
- 2. Table of Contents Part 1: Problems, Communication, and Event-Driven basics Part 2: Event Modelling, Schemas, & Bootstrapping Domain Data Part 3: Service Modelling, Using the DCL, & Examples
- 3. System Architectures It’s all tradeoffs
- 6. (Near-) Real Time updates
- 7. Coupling
- 8. How do we handle these issues in the real world?
- 9. First, let’s look at Communication Structures
- 10. Communication Structures • Business • Implementation • Data
- 11. Business Communication Structure What do we do to achieve our business goals?
- 15. Implementation Communication Structure How do we optimize the processes that lead to our business goals?
- 17. Data Communication Structures How do we get the data we need for our business processes?
- 19. The Implementation Communication Structure Also, The Data Communication Structure!
- 20. Ad-hoc Data Communication Structures
- 21. Synchronous Services can be used to help solve these problems
- 23. Fan-Out
- 24. Scaling Issues
- 25. Boundary & Implementation Issues
- 26. Where do Synchronous Services work well?
- 27. Many companies succeed with synchronous services! It’s all about tradeoffs
- 29. Events are Business Facts
- 30. Example E-commerce • Stores, Merchants, Products, Users, Accounts Shipping • Warehouses, Orders, Shipments, Drivers, Trucks
- 31. Product A has 100 units of stock User A has placed an order, ID=1234 Merchant 100’s minimum order amount is $50 Publish the Facts
- 33. The Event-Broker stores all events and event streams
- 34. The Event Stream (Immutable Log)
- 35. Partitions for Scalability and Data Locality
- 42. Principle Treat domain event data as a first-class citizen
- 43. Does this mean everything should be written to an event-stream? No
- 44. Event Types & Streams 1) Entity 2) Keyed Event 3) Unkeyed Event
- 45. 1) Entity VIN MAKE MODEL COLOUR A1 Ford F150 Tan B2 Toyota Camry Gold Cars ID FIRST LAST 123 Adam Bellemare 444 Guy Incognito People Key Value An entity has a unique key VIN is Key
- 46. Materialize Entities into Each Service Each event is the current state of a (keyed) entity! Partition 0 Overwrite “9” with “2”
- 47. 2) Keyed Events VIN INFRACTION AMOUNT DATE DRIVER_ID A1 Speeding $150 2018-10-07 123 A1 Parking $25 2018-11-11 123 B2 Parking $25 2018-11-13 444 Tickets Issued Key Value Multiple events with same VIN
- 48. 2) Keyed Events VIN INFRACTION AMOUNT DATE DRIVER_ID A1 Speeding $150 2018-10-07 123 A1 Parking $25 2018-11-11 123 B2 Parking $25 2018-11-13 444 Tickets Issued Key Value Multiple events with same VIN “This ticket belongs to VIN ID X”
- 49. Aggregating State from Keyed Events Keyed by Shape Partition 0 Partition 1 Instance 1 Instance 0 eg: Total Ticket Costs
- 50. 3) Unkeyed Events LICENSE PLATE CAMERA_ID DATETIME IMAGE_URI ZXJ123 123 2020-07-07… s3://… ABC123 234 2020-07-08… hdfs://… ACBZ900 345 2020-07-08… c:/program… Intersection Traffic Camera Value
- 51. 3) Unkeyed Events LICENSE PLATE CAMERA_ID DATETIME IMAGE_URI ZXJ123 123 2020-07-07… s3://… ABC123 234 2020-07-08… hdfs://… ACBZ900 345 2020-07-08… c:/program… Not that Common - Usually have a key! Value May be found in “dumb” data pipelining
- 52. A Simple Enrichment Example Cars Tickets Ticket $ per Car 1) Materialize 1) Aggregate 2) Join 3) Emit
- 53. Break
- 54. Part 2: Event Modelling, Schemas, & Bootstrapping Domain Data
- 55. “The fundamental problem of communication is that of reproducing at one point, either exactly or approximately, a message selected at another point.” - Claude Shannon, Father of Communication Theory
- 56. Modelling Events using Domain-Driven Design
- 57. Modelling Events What business occurrence does this event represent?
- 58. Event Modelling Example SALES FACTS FK ITEM_ID FK STORE_ID PRICE PAYMENT ITEM DIMENSIONS PK ITEM_ID NAME BRAND CATEGORY STORE DIMENSIONS PK STORE_ID STORE_NAME ADDRESS CATEGORY Entity Stream Store Stream Key: Store_Id Event Stream Sales Stream Key: N/A Entity Stream Item Stream Key: Item_Id
- 59. Event Modelling Example - Alternative SALES FACTS PK SALES_ID FK ITEM_ID FK STORE_ID PRICE PAYMENT ITEM DIMENSIONS PK ITEM_ID NAME BRAND CATEGORY STORE DIMENSIONS PK STORE_ID STORE_NAME ADDRESS CATEGORY Entity Stream Store Stream Key: Store_Id Entity Stream Item Stream Key: Item_Id Sales Stream Key: Sales_Id Entity Stream
- 60. Event Modelling Example - Denormalized SALES FACTS PK SALES_ID FK ITEM_ID FK STORE_ID PRICE PAYMENT ITEM DIMENSIONS PK ITEM_ID NAME BRAND CATEGORY STORE DIMENSIONS PK STORE_ID STORE_NAME ADDRESS CATEGORY Enriched Sales Stream Key: Sales_Id Merged Entity Stream Join Join
- 61. Unstructured Data A recipe for disaster
- 63. Real-life bloopers isScreenOn: “yes” “no” 1 0 true false null productId: null 1234 “null” “<null>” “%3cnull%3e” premium: true false 4040 null “null”
- 64. Consumer Pain
- 65. Schemas ID (INTEGER) COLOUR (NULL, STRING) DEFAULT: NULL TIME (DATETIME) 100 Red 2020-08-08… 200 NULL 2020-08-08…
- 66. Producer Responsibility: Create events that match the schema
- 67. Schema Options • Apache Avro • Google’s Protobuf • JSONSchema (not plain JSON!)
- 68. (Almost) self-documenting { "type" : "record", "name" : "User", "namespace" : "com.example", "fields" : [ {"name": "username", "type": “string"}, {"name": "phone", "type": [null,"int"], "default" : null}, {"name": "address", "type": "string", "default": “UNKNOWN"}, {"name": "country", "type": "enum", "symbols": ["CA", "US", "OTHER"], "doc" : "The user's country. Select other if not in list"} ] }
- 69. Schema Evolution
- 70. Consuming & Converting Events
- 71. Backwards Compatible Can convert a V2 event to V3 format at Consumer
- 72. Forwards Compatible Can convert a V3 event to V2 format at Consumer
- 73. Full Transitive
- 75. Schema Registry
- 76. Schema Registry
- 79. Breaking Changes New Streams New offsets Migrating consumers Reprocessing
- 82. Criteria • Performance & Throughput • Indefinite retention • Tiered storage • Security & Access Control • Adoption & Open Source • 3rd-party options and tools
- 84. Indefinite Retention It’s just bytes on disk!
- 85. Tiered Storage Put those bytes in the Cloud!
- 86. Security and Access Control • Secure / Encrypted connections • Data Encryption (Field Level Encryption!) • Access Control Lists (ACLs) • Enforce single-writer principle • Track stream ownership and subscriptions
- 87. Adoption & Open Source • Avoid vendor lock-in • Hire talent • Fix own issues • Extended network
- 88. 3rd Party Options & Tools • Schema Registry • Stream Access & Visualization Tooling • Change-Data Capture Tooling • Hosting
- 89. Options Apache Kafka Apache Pulsar AWS Kinesis Azure Event Hub Google Pub Sub Limited Retention (2-52 Weeks) Small Community Tiered Storage Tiered Storage (OSS & Private)
- 90. Populating Event Streams for Data Communication
- 91. What is it that we’re willing to do to provide reliable domain data? Principle: Use the event-broker as the single source of truth
- 94. Forked-Write Anti-Pattern Can’t write to both data stores atomically!
- 99. Forked-Write Anti-Pattern What else can we do?
- 100. Transactional Outbox
- 101. Transactional Outbox
- 102. Transactional Outbox
- 103. Transactional Outbox Works! But can be challenging to set up!
- 104. Pattern: Materialize after Write
- 105. Pattern: Materialize after Write
- 106. Question Won’t these options slow down the application?
- 107. Liberating Data from Legacy Systems
- 108. Connectors & Change-Data Capture
- 109. Connectors & CDC Pros Flexible Scalable Single Source of Truth
- 110. Connectors & CDC Cons Idiosyncracies Overhead Performance Impact Coupling on Internal Data Model Brittle & Reactive
- 111. It’s not one or the other! Point-to-Point or Event-Driven?
- 112. Event Streams are part of the API
- 113. Questions?
- 114. Part 3: Service Modelling, Using Event Streams, & Examples
- 115. The Microservice Tax
- 117. Paying the tax lets your users focus on building the services
- 118. How do you model your services? Use Domain-Driven Design (again)
- 119. Modelling Services Build services around business functions
- 120. Consider the Modular Monolith
- 122. Modelling Services It’s an art as much as a science
- 123. Eg Workflows built from Services
- 124. Let’s look at a subset
- 125. Payment Processing
- 126. Payment Processing
- 127. Payment Processing KEY VALUE OrderId List(ItemId) CustomerInfo KEY VALUE OrderId List(ItemId) CustomerInfo PaymentResults KEY VALUE OrderId PaymentFailureInfo PaymentAPIInfo
- 128. Payment Processing KEY VALUE OrderId List(ItemId) CustomerInfo KEY VALUE OrderId List(ItemId) CustomerInfo KEY VALUE OrderId PaymentInfo KEY VALUE OrderId PaymentResults
- 129. Service Decides Payment Failure Try 3 times Failed 3 times
- 130. Payment Processing: Internal State
- 131. Payment Processing: Internal State
- 134. Reduce Complexity with Field-Level Encryption
- 135. Reduce Complexity with Field-Level Encryption KEY VALUE OrderId List(ItemId) CustomerInfo //Encrypted PaymentInfo (CC, Address, etc) Only Authorized Services can Decrypt the Fields!
- 136. Reduce Complexity with Field-Level Encryption KEY VALUE OrderId List(ItemId) CustomerInfo //Encrypted PaymentInfo (CC, Address, etc) KEY VALUE OrderId PaymentFailureInfo PaymentAPIInfo //Encrypted PaymentResults Only Authorized Services can Decrypt the Fields!
- 138. Expanded
- 139. Schemas KEY VALUE OrderId List(ItemId) CustomerInfo KEY VALUE ItemId Map(Location,Quantity) KEY VALUE OrderId Location FulfimentInfo List(ItemId) CustomerInfo
- 140. Fulfillment Workflow Pt.1 (Automated) High-Priority Materialization!
- 142. Unfulfilled?
- 143. Compensation KEY VALUE FulfimentId Location OrderId List(ItemId) CustomerInfo KEY VALUE OrderId State List(ItemId) CustomerInfo
- 145. In Conclusion…
- 146. Event streams as the DCL
- 147. Event Streams: The Data Communication Layer
- 149. Services Align with Business
- 151. Questions, please!