Microservices, Events, and Breaking the Data Monolith with Kafka
8 likes2,172 views
The document outlines a webinar discussing microservices and event-driven architectures, led by Cornelia Davis and Ben Stopford. Key points include the benefits and challenges of microservices, the importance of resilience in system design, and emerging patterns for database management in microservices. The discussion emphasizes event sourcing, materialized views, and the evolving landscape of software systems towards event-driven architectures.
Microservices, Events, and Breaking the Data Monolith with Kafka
- 1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Version 1.0 Cornelia Davis Vice President, Technology, Pivotal @cdavisafc February 2019 Microservices, Events, and Breaking the Data Monolith Webinar Ben Stopford Office of the CTO, Confluent @benstopford
- 2. Cornelia Developer (wasn’t Ops) Web architectures for >10 years Cloud-native for 6+ years Cloud Foundry for 6+ years Discount code 40% off!: 40cloudnat https://www.manning.com/books/cloud-native @cdavisafc
- 3. Ben Stopford • Leads Office of the CTO team at Confluent. • 20 years in engineering, mostly distributed data. • Worked on Kafka’s replication protocol, rebalancing, throttling. Download book: bit.ly/benstopford @benstopford
- 4. @cdavisafc Where we’re going today Microservices Goodness Microservices Challenges How Event-driven Approaches Help
- 5. Why Microservices ● Scale Applications ● Scale Teams ● Independent Development Cycles ● Experimentation ● Resilience
- 8. Assumption: Each microservice boasts 99.999% availability (and there are 100 of them) Assumption: The network is reliable Result: Home Page achieves 99.9% availability https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
- 9. How we try to add a level of resilience?
- 10. Retries ▸ Client must consider failure ▸ Decide on fall-back behavior ▸ Likely including retries ▸ But then we need to handle downstream consequences of these (retry) behaviors CLIENT SERVICE Timeouts? If we don’t hear back, try again
- 11. But… CLIENT SERVICE CLIENT CLIENT CLIENT ▸ Client must consider failure ▸ Decide on fall-back behavior ▸ Likely including retries ▸ But then we need to handle downstream consequences of these (retry) behaviors
- 12. Circuit Breakers ▸ Client must consider failure ▸ Decide on fall-back behavior ▸ Likely including retries ▸ But then we need to handle downstream consequences of these (retry) behaviors CLIENT SERVICE CLIENT CLIENT CLIENT
- 13. But then, is stale data better than no data? - often it is ▸ Client must consider failure ▸ Decide on fall-back behavior ▸ Likely including caches ▸ But then we need to handle things like cache expiry SERVICE APP SERVICE APP APP SERVICE APP SERVICE APP SERVICE APP
- 15. New LIGHTWEIGHT ARCHITECTURES are emerging Microservices addressing speed to market and cloud scale Monolithic / Layered Microservices
- 17. Pattern: Bounded Context ● Domain-Driven Design ● Each bounded context has a single, unified model ● Relationships between models are explicitly defined ● A product team usually has a strong correlation to a bounded context ● Ideal pattern for Data APIs - do not fall into the trap of simply projecting current data models
- 18. Pattern: Database per Service ● Supports Polyglot persistence ● Independent availability, backup/restore, access patterns, etc.
- 19. Pattern: Database per Service ● Supports Polyglot persistence ● Independent availability, backup/restore, access patterns, etc. But then what about… ● Overlapping data ● Data ownership ● Data synchronization
- 20. More patterns to the rescue
- 21. Request/Response - root to leaf; Events - leaf to roots
- 22. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites ! Response " Response # Response $ Request %Request &Request Connections API User: - id - username - name Connections: - id - follower - followed Posts API Posts: - id - date - userId - title - body Starting with the familiar (Root to Leaf)
- 23. % $ ! & " # Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Response Response Response Request Request Request Connections API User: - id - username - name Connections: - id - follower - followed Posts API Posts: - id - date - userId - title - body Layer in caching to improve performance Cache How do we handle out of data data?
- 24. µService Event Consumer µService Event Producer event store … … Introduce a broker/event store to decouple microservices Key Properties: - Messaging - Scalability/HA - Event Storage & Replay - Stream Processing
- 25. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request 1 2 event store … … 1 Turn the flow on its head (leaves to root) 1 Connection Created Event - id - follower - followed 1 Post Created Event - id - date - userId - title - body Cache (Materialized View) When the request comes in, the Connections’ Post service already has the answer!
- 26. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request 1 1 1 1 2 event store … … Cache is populated when things change, not on request Materialized View View is formed to match what we need: “New from your Network”
- 28. Event Sourcing
- 29. Traditional systems use mutable state DB This isn’t wrong, it’s just lossy
- 30. Events record the user’s journey Shopping Cart Events 2 Trousers added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
- 31. Stored as a stream Stored statefully (think DB) 12.42 12.44 12.49 12.50 12.59 Information lost! Event User Journey
- 32. 12.42 12.44 12.49 12.50 12.59 We can derive the current state (but not the other way around) DERIVE Stream Processor
- 33. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request event store … … Our “Materialized View” is a form of Event Sourcing View Cache the view somewhere convenient Define the view in our code
- 34. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request event store … … When the applications starts it loads the cache from the log View
- 35. Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites event store … … Event Store becomes the Source of Truth Service A Service E C Service D DB Apps Search NoSQL Apps Apps S T R E A M I N G P L A T F O R M Service C DB Apps Search NoSQL Apps App DWH Hadoop S T R E A M I N G P L A T F O R M C Service B DB Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L A T F O R M
- 36. Event Streaming
- 37. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request event store … … Event Streaming View Recommendations Service 1 2 Recommendation Service plugs right into the existing flow
- 38. event store … … Event Streaming Recommendations Service KafkaStreamsAPI Posts Connections Page Views by User Recommendation Kafka Streams wraps up the . solutions to these problems in a single API: - Joins real-time streams - Tables (from streams) - Summarises streams
- 39. Connections’ Posts API Abundant Sunshine Your source for healthy cooking New From Your Network Seasonal Favorites Connections API Posts API Response Request event store … … Event Streaming View Recommendations Service Create ‘views’ derived from events in the log Build event streaming services using KStreams API Decouple services from one another using the event store
- 40. Summing it up…
- 41. Evolution of software systems App AppApp App AppApp App AppApp App AppApp App Kafka Monolith Distributed Monolith Microservices Event-Driven Microservices Microservices or? AppApp App AppApp App
- 44. AppApp App AppApp App Kafka Event-Driven Microservices
- 45. Service Bindings AppApp App AppApp App Kafka Event-Driven Microservices Service Bindings
- 46. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Version 1.0 Cornelia Davis Vice President, Technology, Pivotal @cdavisafc February 2019 Microservices, Events, and Breaking the Data Monolith Webinar Ben Stopford Office of the CTO, Confluent @benstopford
- 47. Join us! Keynote Speaker: James Watters Senior Vice President of Pivotal’s Product and Business Development Organizations Register now at Kafka-Summit.org, and get a special 25% off using this code: KSNYPivotal25