Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and gRpc - Codemotion
- 1. natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil Polyglot, Fault-Tolerant, & Event-Driven Programming with Kafka, Kubernetes and gRPC Natan Silnitsky Backend Infra Developer, Wix.com
- 2. registered users from 190 countries 200M of all internet websites run on Wix 5% About Wix @NSilnitsky
- 5. Wix 1,700M Kafka messages a day @NSilnitsky
- 6. Wix 1,700M Kafka messages a day So, we need our message flows to be performant, fault-tolerant, and polyglot.
- 7. Agenda Event-driven programming with Kafka + Performance with Greyhound + Fault-tolerance with Greyhound & Kubernetes + Polyglot with Kubernetes & gRPC
- 8. @NSilnitsky HTTP * coupled Request-Reply Communication New App installed Site Apps Service ECom Catalog Service Classic.
- 9. @NSilnitsky HTTP New App installed What if Network is Unreliable Request-Reply Communication Site Apps Service ECom Catalog Service
- 10. @NSilnitsky * 2000 bottle HTTP New App installed Request-Reply Communication Cascading Failures can happen.
- 11. @NSilnitsky Message Consumer Message Producer Broker Event-driven Communication Introduce a Broker. * clusters and replications
- 12. @NSilnitsky Broker Event-driven Communication Message Consumer Message Producer Introduce a Broker.
- 13. @NSilnitsky Kafka Broker Event-driven Communication Site Apps Topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Where you store messages (events!) A Kafka Broker Looks Like This: * perf vs queues
- 14. @NSilnitsky Kafka Broker Event-driven Communication Site Apps Topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Kafka Producer Site Apps Service There’s the Event Producer,
- 15. @NSilnitsky Kafka Broker Event-driven Communication Site Apps Topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Kafka Consumer Ecom Catalog Service and Event Consumer. * scale
- 16. Site Apps Service Kafka Consumer Kafka Producer Greyhound wraps Kafka Ecom Catalog Service Kafka Broker Greyhound Producer Greyhound Consumer * open source
- 17. @NSilnitsky Simplify APIs, with additional features Greyhound wraps Kafka Site Apps Service Kafka Consumer Kafka Producer Ecom Catalog Service Kafka Broker
- 18. @NSilnitsky ✔ Parallel Consumption ✔ Consumer Retry Strategies ✔ Resilient Producer ✔ Context Propagation ✔ Metrics reporting ✔ Batch Consumer - Soon
- 19. @NSilnitsky Abstract so that it is easy to change for everyone Kafka Consumer Kafka Producer Kafka Broker Greyhound wraps Kafka
- 20. Performant Event-driven Programming with Kafka and Greyhound (Wix OSS) Wix 1,700M Kafka messages a day
- 21. @NSilnitsky Kafka Broker Topic Greyhound Consumer Kafka Consumer 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Performant message handling A Service Message Handler
- 22. @NSilnitsky Kafka Broker Topic Greyhound Consumer Kafka Consumer SCALA ZIO FIBERS + QUEUES 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 (Thread-safe) Parallel Message Consumption Performant message handling A Service Message Handler
- 23. @NSilnitsky 80 partitions Kafka Broker Topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Performant message handling Maximum Throughput: Messages Processing - Non blocking IO (Netty grpc client) with response latency of 100ms
- 24. @NSilnitsky 80 partitions Kafka Broker Topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Performant message handling Maximum Throughput: 800 Messages per second Messages Processing - Non blocking IO (Netty grpc client) with response latency of 100ms
- 25. @NSilnitsky Kafka Consumer Kafka Consumer Kafka Consumer Kafka Consumer Kafka Consumer Kafka Consumer Kafka Consumer Performant message handling 80 Kafka Consumers with 80 Java threads 80 partitions Kafka Broker Topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Messages Processing - Non blocking IO (Netty grpc client) with latency of 100ms Kafka Consumer
- 26. @NSilnitsky Greyhound Performant message handling or 1 Greyhound Consumer with 80 fibers running on a small thread pool Kafka Consumer 80 partitions Kafka Broker Topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Messages Processing - Non blocking IO (Netty grpc client) with latency of 100ms
- 27. fault-tolerant Event-driven Programming with Kafka, Greyhound & Kubernetes Wix 1,700M Kafka messages a day
- 28. @NSilnitsky Kafka Broker renew-sub-topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Greyhound Consumer Kafka Consumer Fails To process Fault-tolerant message handling
- 29. @NSilnitsky renew-sub-topic 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 renew-sub-topic-retry-0 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 renew-sub-topic-retry-1 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 Greyhound Consumer Kafka Consumer RETRY PRODUCER Fault-tolerant message handling Inspired by Uber RETRY! Kafka Broker
- 30. @NSilnitsky Producer Wix Payments Service Subscription renewal Job Scheduler Fault-tolerant message handling Use Case: Guarantee Completion Kafka Broker
- 31. @NSilnitsky + Retry on failure Kafka Broker Producer Consumer Wix Payments Service Subscription renewal Job Scheduler Fault-tolerant message handling Use Case: Guarantee Completion
- 32. @NSilnitsky Producer Consumer Subscription renewal Job Scheduler Fault-tolerant message handling Wix Payments Service Use Case: Guarantee Completion Kafka Broker + Retry on failure
- 33. @NSilnitsky Producer Fault-tolerant message handling The Resilient Producer Kafka Broker
- 34. @NSilnitsky Save message to disk Kafka Broker Producer Fault-tolerant message handling The Resilient Producer: When failed to produce will save the message
- 35. @NSilnitsky Kafka Broker Producer Fault-tolerant message handling The Resilient Producer: When failed to produce will save the message and retry on failure. + Retry on failure
- 36. @NSilnitsky Kebe Fault-tolerant message handling What if pod is killed? pod Kube Node
- 37. @NSilnitsky Kafka Broker pod Scavenger Kube Node DaemonSet What if pod is killed? Fault-tolerant message handling Flush out messages * small footprint
- 38. Polyglot Event-driven Programming with Kubernetes & gRPC Wix 1,700M Kafka messages a day
- 39. @NSilnitsky Polyglot message handling Motivation: code reuse Kafka Broker Greyhound Scala/Java services Greynode NodeJS services Producer Consumer Producer Consumer
- 40. @NSilnitsky Polyglot message handling Partial implementation is a problem Kafka Broker Greyhound Greynode NodeJS services Producer Consumer Producer Consumer
- 41. @NSilnitsky Greyhound Polyglot message handling Greyhound Sidecar Kafka Broker NodeJS services Consumer gRPC Producer
- 42. @NSilnitsky Kafka Broker Greyhound Polyglot message handling Greyhound Sidecar gRPC .proto Sidecar written with Scala Service written with JS & TS
- 43. @NSilnitsky Kafka Broker Greyhound Polyglot message handling Greyhound Sidecar gRPC Sidecar written with Scala .proto Service written with Python gRPC Service written with JS & TS
- 44. 44 @NSilnitsky Polyglot message handling pod Kube Node Containerized app Volume The Sidecar resides with the app in the same pod.
- 45. @NSilnitsky Kubernetes pod with main and sidecar containers Polyglot message handling pod Kube Node pod pod Kafka Broker @NSilnitsky
- 46. @NSilnitsky There’s a memory issue. pod Kube Node pod pod Polyglot message handling Kafka Broker @NSilnitsky
- 47. @NSilnitsky Kafka Broker DaemonSet Optimization: Greyhound in Daemonset Polyglot message handling @NSilnitsky
- 48. So, We use Kubernetes’ flexibility to deploy Greyhound producers and consumers in different patterns, in order to comply with different requirements. @NSilnitsky
- 49. Wix harnesses Kafka, Kubernetes and gRPC to achieve a polyglot, fault tolerant, scalable event-driven distributed system.
- 50. @NSilnitsky A Java/Scala high-level SDK for Apache Kafka. 0.1 is out! github.com/wix/greyhound
- 51. @NSilnitsky Thank You natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil