Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov and Michael Ng, Confluent) Kafka Summit NYC 2019
3 likes1,702 views
The document discusses deploying Kafka on Kubernetes, highlighting challenges such as security configuration and external access. It emphasizes the use of Confluent's Helm charts, StatefullSets, and the need for operator expertise to effectively manage the platform. The session also outlines the progress of their operator's private preview phase with notable customers testing the solution.
Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov and Michael Ng, Confluent) Kafka Summit NYC 2019
- 1. @gamussa | #kafkasummit | @ConfluentINc Kafka on Kubernetes: Does it really have to be «The Hard Way»? April, 2019 / New York, 2019 @gamussa | #kafkasummit | @ConfluentINc
- 2. Raffle, yeah 🚀 Follow @gamussa @confluentinc 📸 🖼 👬 Tag @gamussa With #kafkasummit
- 3. @gamussa | #kafkasummit | @ConfluentINc 3 Evolution of #devkafkaops Shell scripts ansible/chef Docker Kubernetes
- 4. @gamussa | #kafkasummit | @ConfluentINc 4
- 5. @gamussa | #kafkasummit | @ConfluentINc 5 🙋
- 6. @gamussa | #kafkasummit | @ConfluentINc 6 Who run stateless workloads in Kubernetes? Who thinks it’s a good idea? Who run stateful workloads in Kubernetes? Who thinks it’s a good idea? 🙋
- 7. @gamussa | #kafkasummit | @ConfluentINc 7 kafkaesque world of Kafka on Kubernetes
- 8. @gamussa | #kafkasummit | @ConfluentINc 8 Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades #devkafkaops
- 9. @gamussa | #kafkasummit | @ConfluentINc 9 We just need to deploy Kafka on Kubernetes
- 10. @gamussa | #kafkasummit | @ConfluentINc 10 We will use confluentinc/cp-helm- charts
- 11. @gamussa | #kafkasummit | @ConfluentINc 11 Helm Charts is just a GO Templates. How Charts help with rolling restart?
- 12. @gamussa | #kafkasummit | @ConfluentINc 12 We will use StatefulSets with OrderedReady
- 13. @gamussa | #kafkasummit | @ConfluentINc 13 We need SRE / Operator knowledge to manage the platform. You need Operator!
- 14. @gamussa | #kafkasummit | @ConfluentINc 14
- 15. @gamussa | #kafkasummit | @ConfluentINc 15 Show me your Operator
- 16. @gamussa | #kafkasummit | @ConfluentINc 16 Demo
- 17. @gamussa | #kafkasummit | @ConfluentINc 17 DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MIND
- 18. @gamussa | #kafkasummit | @ConfluentINc 18 What just happened? ZK and Kafka deployed Security with TLS is configured External access is configured Monitoring is enabled
- 19. @gamussa | #kafkasummit | @ConfluentINc 19 Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components
- 20. @gamussa | #kafkasummit | @ConfluentINc 20 Confluent Operator - Scale Automate Scaling: Spin up new brokers, connect workers easily Distribute partitions to new brokers: Determine balancing plan Execute balancing plan Monitor Resources
- 21. @gamussa | #kafkasummit | @ConfluentINc 21 Be like Justin!
- 22. @gamussa | #kafkasummit | @ConfluentINc 22 Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under- replicated partitions 5. Upgrade the next broker
- 23. @gamussa | #kafkasummit | @ConfluentINc 23 Will it fly? vs.
- 24. @gamussa | #kafkasummit | @ConfluentINc 24 GA Plans● We are in private Preview Release now ● 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech ● We are in the final stages of Preview and about to launch soon
- 25. @@gamussa | #kafkasummit | @ConfluentINc Thanks! @gamussa viktor@confluent.io michael.ng@confluent.io https://slackpass.io/confluentcommunity #kubernetes
- 26. 26