Topic and schema management-meetupberlin
1 like267 views
The document discusses key topics related to building a streaming platform using Apache Kafka, including topic and schema management, creating self-sufficient teams, and database migrations. It highlights common management questions for Kafka operations teams and suggests methods for organizing and replicating state across environments. Various tools and approaches, such as CI/CD pipelines and web interfaces, are recommended to achieve team autonomy and effective deployment in Kafka systems.
Topic and schema management-meetupberlin
- 1. 1 Wax on Wax off: The Learnings of Karate kid applied to Apache Kafka Pere Urbon-Bayes (@purbon) Solutions Architect
- 2. 2 @purbon Topics for today ● Common questions when building an Streaming Platform with Apache Kafka? ○ Topic Management ○ Schemas Management ● Building self sufficient teams with the Kafka Topology Builder ● Database migrations for Apache Kafka, the Kafka Migration tool
- 3. 33 Common management questions for KafkaOps teams
- 5. 5 @purbon Topics with structure and automation!
- 6. 6 @purbon Automate all the things!
- 8. 8 @purbon A CI/CD pipeline for Kafka Management
- 12. 12 @purbon An imperative CI/CD pipeline
- 13. 13 @purbon But not only for CI/CD
- 14. 14 @purbon
- 15. 15 @purbon
- 16. 16 @purbon
- 17. 17 @purbon But how do we organise this How do you replicate state between environments: ● Upgrade changes across environments. ○ Merge changes across descriptors. ○ Apply migrations across environments. ● Graduate state from lower environments. ○ Useful for schema registry using the version/latest Resource ○ Very hard for Apache Kafka (pulling latest state is very hard) ● State replication needs CI/CD and automation.
- 18. 18 @purbon As teams using Apache Kafka As a user of an Apache Kafka based system, I want to: ● Be self sufficient ● Care about my area of work / project ● Get test and validation for all the changes requested
- 19. 19 @purbon Approaches for making self sufficient teams There are different approaches to make self sufficient teams: ● Via CI/CD and github. ● Using a web interface (ala Control Center, Kafkawize or KafkaTool) ● Using service brokers / operators (ala Containers)
- 20. 2020 Questions? Thanks! Pere Urbon Bayes Solution Architect pere@confluent.io