Microservices on top of kafka
Download as PPTX, PDF1 like2,291 views
1) The document discusses how a company gradually transitioned from a monolithic architecture to microservices using Apache Kafka as the backbone. 2) It outlines the steps taken, including defining service responsibilities and data models, using event sourcing and CQRS patterns, designing Kafka topics, and validating data. 3) The document emphasizes that Kafka should be the single source of truth for critical data and applications should be able to reprocess historical data from Kafka topics.
Microservices on top of kafka
- 1. Microservices on top of Kafka How we got lost during the transition to Microservices, and how we found our way out using Kafka Vladi Feigin Software Architect, LivePerson vladif@liveperson.com
- 2. “This is an attempt to share our experience in breaking a monolith and moving to a microservices world” … Using Kafka Preface
- 3. Active customers Mission critical application Complex business logic Very high uptime SLA The Monolith
- 4. The Decision
- 6. Develop a new system from scratch OR Gradually break the existing monolith We believe the second option is the way to go! How can it be broken?
- 7. Monolith First app out (shadow mode) Monolith First app out Monolith without app Gradual progress Events to Kafka. Tests! App out in shadow mode. Tests! App out in production mode The Gradual Approach
- 8. TODO List Microservices on Kafka
- 9. Define every service responsibility Design services behavior first (the actions it does) Define Data Model (see next slide) Apply Domain Driven Design principles Revise Data Model
- 10. Command (Request) is an action, addressed to, and consumed by, one single service Fact is a declaration to the rest of world about the service state change (triggered by a command). Entities - the business objects Define Commands, Facts and Entities
- 11. A schema defines the language the services use to talk to each other It can be JSON, Avro or Protobuf Define and enforce rules for schema compatibility Every schema change must be validated Use schema
- 13. Separate the write from the read operations Facts written chronologically into dedicated Kafka (event-sourced) topics Create read-optimized views in external DB for queries Use Event Sourcing and CQRS
- 14. Every service is allowed to write only on the topics it owns Services are allowed to read from other service topics Follow Single Writer Principle
- 15. Service topics should reflect the service data model Topics are integral to all business flows Every service usually has its own topic Topic design is similar to the designing DB table Carefully design Kafka topics
- 16. ● Validate events before you write them to Kafka ● Validate against schema ● Validate against local state Validate, Validate, Validate!
- 17. Data crucial for managing the core business flows should be managed in Kafka Avoid having other “moving parts”, such as a database for managing the core business flow Make your critical data mobility simple Kafka is Single Source of Truth
- 18. Your application should be able to reprocess historical data from Kafka This means reprocessing facts executed from the event-sourced topics Design for reprocessing
- 19. Monitor You must have full visibility over what’s going on in the system, from the very beginning of the process Constantly look for unexpected behaviour and anomalies
- 20. Be a smart pessimist Be prepared for non-happy and edge case scenarios! List all possible difficult scenarios and use your architecture to test them Ensure you have a solution for every scenario
- 21. Postscript Microservices Cons: ● Microservices is hard and challenging ● Operational costs are high ● Hard to debug in production Microservices Pros: ● Unleash development speed ● Services scale-out is easier ● Clear roles and responsibilities of services
- 22. LivePerson is hiring! In Tel-Aviv and Raanana https://www.liveperson.com/company/careers We’re hiring
- 23. Appendix
- 24. Consider Kafka Compacted topics Long topic retention Deletion by fact key Smaller local state Custom retention mechanisms The only option for supporting GDPR
- 25. Producer parameters: ● acks=all ● Many retries on failure If events order is critical set : ● max.in.flight.requests.per.connection=1 If your data throughput allow - use synchronous send Server configuration: ● unclean.leader.election.enable=false ● min.isr=2 ● replication factor at least 3 Configuration