The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Summit NYC 2019
4 likes2,149 views
The document discusses the transition from monolithic architecture to event-driven microservices in the Flipp app, used by millions for shopping deals. It emphasizes principles like a single source of truth and the use of Kafka for data streaming, while outlining a modular approach to application development. Key topics include system decoupling, bounded contexts, data liberation, and the challenges and strategies related to managing microservices and core events.
The Migration to Event-Driven Microservices (Adam Bellemare, Flipp) Kafka Summit NYC 2019
- 2. The Flipp app is used by millions of people each week to find deals and plan their shopping 50M Downloads 90% of North America’s leading retailers
- 3. Agenda 1. Background 2. Principles 3. Liberating Data 4. Building Event-Driven Microservices 5. Core Events
- 5. Classic Monolithic Architecture Monolith sprawl Teams sharing ownership (slow) Tangled web of dependencies Significant Technical Debt Existing implementations hindering change Multiple sources of truth
- 6. Moving Forwards Modularize and decompose monoliths While… Event-driven microservice • Decouple systems, teams and products • Streamline application development
- 8. 1 - Single Source of Truth Kafka
- 9. 2 - Use Event-Driven Microservices MicroserviceTopic Topic Microservice MicroserviceTopic REST API Request/Response
- 10. 3 - Avro Schemas • Confluent Schema Registry • Avro-encode everything published to Kafka • Compatibility & Isolation Schema Registry Producer ConsumerTopic 1) Validate / Register 2) Publish 3) Consume 4) Obtain Schema 5) Process
- 11. 4 - Bounded Contexts - Services Microservice Topic - User Images Business Functions Technological Functions Topic - Partner Microservice File Store Technological Functions Microservice File Store Business Functions Microservice Topic - User Images Topic - Partner
- 12. 5 - Align Systems, Teams and Products Team 1 Team 2 PII Financial Outgoing: 3 Incoming: 0 Outgoing: 0 Incoming: 3
- 13. 6a - Language and Technology Toolbox Free-for-All Fully RestrictedToolbox Criteria & Selection Process Toolbox Languages Frameworks Datastores Services Accepted
- 14. 6b - Toolbox -Templates & Generators ~Topic- Topic- State State ~Topic- REST API ~Topic- Topic- REST API ~ Topic-
- 15. 7 - Single Unit of Deployability State~Topic- REST API Service 1 Service 2 Service 3 Single Unit of Deployability
- 16. 3 - Liberating Data
- 17. Why Liberation • Decouple existing systems • Support single-source of truth Principle • Necessary for EDM
- 18. Kafka Connect - Pros • Easy to get started • Supports many connectors • Can be use DB CDC, like Debezium • Can source and sink data
- 19. Kafka Connect - Cons • Inadvertently expose inner data structures • Performance Issues (Source) • Cluster Overhead • Inter-team dependencies (owners vs users) • Setting Precedence - Lethargy towards moving to EDM
- 20. Direct Production of Records Monolith Tabl e Tabl e Tabl e Tabl e Event Producing Logic Isolate the Internals Topic Topic Topic Uses Confluent Schema Registry
- 21. Monolith - Forked Writer Anti-Pattern Monolith Data Table 1) Update Data 2) Update Table 3) Write to Kafka Topic
- 22. Lesson - Transactions and Output Table ActiveReco rd (Rails) Event Output Table Data Table 2) Transaction 1) Update Data Producer (Separate Thread) 3) Asynchronous Publish Topic
- 23. 4 - Building Event-Driven Microservices
- 24. What does building an app look like? Topics & Schemas 1) Data Discovery
- 25. Data Discovery - Data and Topic Lookup Single Source of Truth
- 26. What does building an app look like? Topics & Schemas 1) Data Discovery 2) Create microservice code repo register microservic 3) Register for R/W Topic permissions Topic-A Topic-B Write Read Read 4) Obtain Schemas + Copy to Code Base Schema-A.avsc Schema-B.avsc code repo 5) Generate Code from Schemas Schema-A.avsc Schema-B.avsc SchemaB.java SchemaB.java 6) Write Business Logic Topic- Topic- Busines s Logic Topic- Topic- CI & CD
- 27. Denormalization + Join Strategy Exposed relational data as streams Exists in our data models Foreign keys Denormalization vs. Joining
- 28. Relational Data - Downstream Join Monolith Tabl e Tabl e Tabl e Tabl e Topic Topic Topic Tabl e Tabl e Topic Topic Consumer Consumer
- 29. Relational Data - Upstream Denormalize Monolith Tabl e Tabl e Tabl e Tabl e Tabl e Tabl e Merge Topic Topic Consumer Consumer
- 30. Denormalization + Join Strategy Denormalize: Closely related properties Commonly referenced properties (Data domain knowledge) Consumers perform joins on: Distant data Verbose data
- 31. Example - Item Availability Items Merchants Many Items to One Merchant Topics Item: Key=123, Value={size:M, color: Red, merchant: 500, … } Merchant: Key=500, Value={name: Josh’s Fishing & Beards, … } Event Examples Stock Stock: Key=123, Value={LocationId:9000, Ordered:500, Online:
- 32. What do we do Items Merchants Many Items to One Merchant Topics Stock One Stock to One Item Item Eventification Service Enriched- Topic Item-Stock Service
- 33. Foreign-Key Joins (Stateful) Kafka Streams (KIP-213 - Used Internally) External Data Store - Relational Database Kafka Streams - Rekey + Group By Key? - Limited by message space - List(A,B,C,D,E,F,G,H,I…) - Reconciling additions and deletions is extremely problematic
- 34. 5 - Core Events
- 35. Core Events Primitives Basic entities (merchants, stock, price, items) Basic events (user clicked on item) Core events Abstractions Based on primitives or other core events Business-wide meaning
- 36. Example - Budget Fulfillment Budget Merchant UserClick Topics Budget Calculation Budget Calculation Budget Calculation App Backend Merchant Reporting Search Team 1 Team 2 Team 3
- 37. Example - Budget as Core Event Budget Merchant UserClick Topics Budget Calculation App Backend Merchant Reporting Search BudgetFill Team 1 Team 2 Team 3 Core Event
- 38. Other Core Event Examples Resource Availability Budget, Stock, Promotion, Order Fulfillment Explicit Patterns of User Behaviour (aka Audiences) Fraud-risk, churn-risk, geo-fencing
- 39. Primitives Core
- 40. Questions?