![@NSilnitsky
500B >600
GAs Per Day
~1500
Developers
Daily HTTP
Transactions
18
Data
Centers &
Pops
2
Cloud Providers
[Google/AWS]
Scaling out
Kafka- based Global Data Mesh](https://image.slidesharecdn.com/devdayseurope-kafkabasedglobaldatamesh-220428084558/85/Dev-Days-Europe-Kafka-based-Global-Data-Mesh-at-Wix-5-320.jpg)
![@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka](https://image.slidesharecdn.com/devdayseurope-kafkabasedglobaldatamesh-220428084558/85/Dev-Days-Europe-Kafka-based-Global-Data-Mesh-at-Wix-32-320.jpg)
![@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka](https://image.slidesharecdn.com/devdayseurope-kafkabasedglobaldatamesh-220428084558/85/Dev-Days-Europe-Kafka-based-Global-Data-Mesh-at-Wix-39-320.jpg)
![@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. ✔ Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka](https://image.slidesharecdn.com/devdayseurope-kafkabasedglobaldatamesh-220428084558/85/Dev-Days-Europe-Kafka-based-Global-Data-Mesh-at-Wix-42-320.jpg)
![@NSilnitsky
DC1 DC2
To where needed
[WIP] Just Produce ❤
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
Master Active](https://image.slidesharecdn.com/devdayseurope-kafkabasedglobaldatamesh-220428084558/85/Dev-Days-Europe-Kafka-based-Global-Data-Mesh-at-Wix-43-320.jpg)
![@NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Master Active](https://image.slidesharecdn.com/devdayseurope-kafkabasedglobaldatamesh-220428084558/85/Dev-Days-Europe-Kafka-based-Global-Data-Mesh-at-Wix-44-320.jpg)
![@NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Master Active
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Replication service
* reintroduce decoupling](https://image.slidesharecdn.com/devdayseurope-kafkabasedglobaldatamesh-220428084558/85/Dev-Days-Europe-Kafka-based-Global-Data-Mesh-at-Wix-45-320.jpg)
Dev Days Europe - Kafka based Global Data Mesh at Wix
- 1. @NSilnitsky @NSilnitsky Kafka- based Global Data Mesh Kafka-based Global Data Mesh at Wix Natan Silnitsky Backend Infra TL, Wix.com natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
- 2. @NSilnitsky Kafka- based Global Data Mesh
- 3. @NSilnitsky @NSilnitsky Kafka- based Global Data Mesh Enable any business, community or person on earth to make their online dream come true. Our mission
- 4. @NSilnitsky Kafka- based Global Data Mesh At Wix ±1B Unique visitors >100M Websites Published by Wix users (5-7% of all internet websites) ±5000 People Work at Wix use Wix platform every month
- 5. @NSilnitsky 500B >600 GAs Per Day ~1500 Developers Daily HTTP Transactions 18 Data Centers & Pops 2 Cloud Providers [Google/AWS] Scaling out Kafka- based Global Data Mesh
- 6. @NSilnitsky Wix has over 2200 microservices in production Kafka- based Global Data Mesh
- 7. @NSilnitsky @NSilnitsky A Fully Distributed Environment 70B daily produced Events 4 Regions @ GCP/AWS ~50k Topics > 500k Partitions 25% of the data is replicated + We assume x10 scale factor during 12-24 next months
- 8. @NSilnitsky @NSilnitsky Growing Pains Large scale traffic Hundreds of developers Event-driven architecture Diverse tech stack Cluster overload DC awareness * keep simple even. Migrated, disributed
- 9. @NSilnitsky @NSilnitsky Growing Pains Large scale traffic Hundreds of developers Event-driven architecture Diverse tech stack Cluster overload DC awareness * messaging platform
- 10. @NSilnitsky @NSilnitsky Growing Pains Large scale traffic Hundreds of developers Event-driven architecture Diverse tech stack Cluster overload DC awareness * tackled blockers
- 11. @NSilnitsky @NSilnitsky Event-driven architecture Kafka Growing Pains Large scale traffic Hundreds of developers Diverse tech stack Cluster overload DC awareness
- 12. @NSilnitsky Kafka Broker You Know Kafka Kafka Consumer Kafka Broker Service A Service B Kafka Producer Kafka- based Global Data Mesh
- 13. @NSilnitsky If You Don’t Know Kafka Kafka- based Global Data Mesh Kafka Broker Kafka Broker Topic Partition
- 14. @NSilnitsky Kafka Broker Topic Kafka- based Global Data Mesh Partition Kafka Producer 0 1 2 3 4 5 Service A
- 15. @NSilnitsky Kafka Broker Topic Kafka- based Global Data Mesh Partition Kafka Producer 0 1 2 3 4 5 Service A Service B Kafka Consumer
- 16. @NSilnitsky @NSilnitsky Large scale traffic Hundreds of developers Event-driven architecture Diverse tech stack Kafka Growing Pains Cluster overload DC awareness
- 17. @NSilnitsky Greyhound Consumer Greyhound Producer Greyhound* Wraps Kafka * open source software Kafka Broker Kafka- based Global Data Mesh Kafka Producer Service A Service B Kafka Consumer
- 18. @NSilnitsky Greyhound Consumer Greyhound Producer Kafka Broker Kafka- based Global Data Mesh Kafka Producer Service A Service B Kafka Consumer
- 19. @NSilnitsky @NSilnitsky aGreyhoundConsumerSpec(groupName, messageHandler, topicName) val producer = GreyhoundBuilder.resilientProducer(topicName) Kafka- based Global Data Mesh
- 20. @NSilnitsky ✔ Parallel Consumption ✔ Batch Consumer ✔ Consumer Retry Strategies ✔ Resilient Producer ✔ Context Propagation ✔ Metrics reporting Self Service 🟊 Greyhound Kafka- based Global Data Mesh Kafka Broker
- 21. @NSilnitsky Scala NodeJS / Python / Go Greyhound Kafka Broker I’m a container gRPC + proto sidecar JVM library Kafka- based Global Data Mesh
- 22. @NSilnitsky Kafka Broker Domain Events ETL to DWH Task Scheduling Async Queue ... We use it for... Kafka- based Global Data Mesh Infra
- 23. @NSilnitsky @NSilnitsky Hundreds of developers Event-driven architecture Diverse tech stack Kafka + Greyhound Growing Pains Large scale traffic Cluster Overload DC awareness
- 24. @NSilnitsky @NSilnitsky DC1 DC2 DC3 DC4 Kafka Broker Single Cluster Overload * intra-dc
- 25. @NSilnitsky @NSilnitsky DC1 DC2 DC3 DC4 Single Cluster Overload A A A A B Kafka Cluster (by SLAs) B Multi Cluster C
- 26. @NSilnitsky @NSilnitsky DC1 DC2 DC3 DC4 A A A A B To which Cluster? Kafka Cluster (by SLAs) B I want to produce a domain event. C
- 27. @NSilnitsky @NSilnitsky aGreyhoundConsumerSpec(groupName, messageHandler, topicName) .ClusterA val producer = GreyhoundBuilder.resilientProducer(topicName, _.ClusterB Kafka- based Global Data Mesh * wix-layer
- 28. @NSilnitsky Cluster A + DC1 Cluster A + DC2
- 29. @NSilnitsky @NSilnitsky Hundreds of developers Event-driven architecture Diverse tech stack Kafka + Greyhound Growing Pains Large scale traffic Cluster Overload DC awareness + Multi Cluster
- 30. @NSilnitsky @NSilnitsky DC1 DC2 DC3 DC4 Master Active Master Passive Read-Only DBs Http/gRPC To which DC? I want to process the event and save the data.
- 31. @NSilnitsky @NSilnitsky Kafka + Greyhound + Cross DC services Kafka- based Global Data Mesh Growing Pains Hundreds of developers Event-driven architecture Diverse tech stack Large scale traffic Cluster Overload DC awareness + Multi Cluster
- 32. @NSilnitsky @NSilnitsky Cross-DC Self Services 🟊 1. Replicate across DCs 2. Produce to Main DC [WIP] Just Produce ❤ Greyhound Kafka
- 33. @NSilnitsky Replicate ! DC1 DC2 across DCs Greyhound producer Greyhound consumer Kafka Broker Cluster A Kafka Broker Cluster Kafka- based Global Data Mesh A
- 34. @NSilnitsky Greyhound Batch Consumer Greyhound Producer Batch # Replicate ! Kafka Broker Cluster A Kafka Broker Cluster DC1 DC2 How? Greyhound producer Greyhound consumer Kafka- based Global Data Mesh A Cross DC Replication Service
- 35. @NSilnitsky Replicate ! DC1 DC2 a Compact Topic for Policy A A Kafka Broker Cluster Kafka Broker Cluster Greyhound producer Greyhound consumer Kafka- based Global Data Mesh
- 36. @NSilnitsky Replication service Replicate ! DC1 DC2 How? A KVStoreReader In-Memory Map Key Value Topic1 AllDCs Topic2 Custom A Kafka Broker Cluster Kafka Broker Cluster Greyhound producer Greyhound consumer Kafka- based Global Data Mesh
- 37. @NSilnitsky Kafka- based Global Data Mesh
- 38. @NSilnitsky Kafka- based Global Data Mesh
- 39. @NSilnitsky @NSilnitsky Cross-DC Self Services 🟊 1. ✔ Replicate across DCs 2. Produce to Main DC [WIP] Just Produce ❤ Greyhound Kafka
- 40. @NSilnitsky Produce ! DC1 DC2 To main DC Read-Only DB / Master Passive DB Master Active A Kafka Broker Cluster Kafka- based Global Data Mesh Greyhound producer
- 41. @NSilnitsky main-dc-producer service Produce ! DC1 DC2 How? Master Active Kafka- based Global Data Mesh Greyhound gRPC producer producer(topicName, _.toMainDc)
- 42. @NSilnitsky @NSilnitsky Cross-DC Self Services 🟊 1. ✔ Replicate across DCs 2. ✔ Produce to Main DC [WIP] Just Produce ❤ Greyhound Kafka
- 43. @NSilnitsky DC1 DC2 To where needed [WIP] Just Produce ❤ Kafka- based Global Data Mesh Greyhound producer Greyhound consumer Master Active
- 44. @NSilnitsky Topology Topic List [consumer group:DC] DC1 DC2 A Kafka Broker Cluster Kafka- based Global Data Mesh Greyhound producer Greyhound consumer To where needed [WIP] Just Produce ❤ Master Active
- 45. @NSilnitsky Topology Topic List [consumer group:DC] DC1 DC2 A Kafka Broker Cluster Master Active Kafka- based Global Data Mesh Greyhound producer Greyhound consumer To where needed [WIP] Just Produce ❤ Replication service * reintroduce decoupling
- 46. @NSilnitsky @NSilnitsky Kafka- based Global Data Mesh DC1 DC2 DC3 DC4 A A A A B B Just Produce ❤ To a Single Logical Kafka Cluster. Kafka Broker Cluster DB
- 47. @NSilnitsky @NSilnitsky Kafka- based Global Data Mesh Any Tech Stack Greyhound Anywhere You Need developer
- 49. @NSilnitsky @NSilnitsky Thank You! natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 👉 slideshare.net/NatanSilnitsky Any questions? Kafka- based Global Data Mesh
- 50. @NSilnitsky @NSilnitsky Q&A natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 👉 slideshare.net/NatanSilnitsky Any questions? Kafka- based Global Data Mesh