SlideShare a Scribd company logo
@NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Kafka-based Global Data
Mesh at Wix
Natan Silnitsky Backend Infra TL, Wix.com
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
@NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* keep simple even. Migrated, disributed
@NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* messaging platform
@NSilnitsky
@NSilnitsky
Growing
Pains
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Cluster
overload
DC
awareness
* tackled blockers
@NSilnitsky
@NSilnitsky
Event-driven
architecture
Kafka
Growing
Pains
Large scale
traffic
Hundreds of
developers
Diverse
tech stack
Cluster
overload
DC
awareness
@NSilnitsky
Kafka Broker
You Know
Kafka
Kafka
Consumer
Kafka Broker
Service A Service B
Kafka
Producer
Kafka- based Global Data Mesh
@NSilnitsky
If You Don’t
Know Kafka
Kafka- based Global Data Mesh
Kafka Broker
Kafka Broker
Topic
Partition
@NSilnitsky
Kafka
Broker Topic
Kafka- based Global Data Mesh
Partition
Kafka
Producer
0 1 2 3 4 5
Service A
@NSilnitsky
Kafka
Broker Topic
Kafka- based Global Data Mesh
Partition
Kafka
Producer
0 1 2 3 4 5
Service A
Service B
Kafka
Consumer
@NSilnitsky
@NSilnitsky
Large scale
traffic
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka
Growing
Pains
Cluster
overload
DC
awareness
@NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Greyhound*
Wraps
Kafka
* open source software
Kafka Broker
Kafka- based Global Data Mesh
Kafka
Producer
Service A Service B
Kafka
Consumer
@NSilnitsky
Greyhound
Consumer
Greyhound
Producer
Kafka Broker
Kafka- based Global Data Mesh
Kafka
Producer
Service A Service B
Kafka
Consumer
@NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
val producer = GreyhoundBuilder.resilientProducer(topicName)
Kafka- based Global Data Mesh
@NSilnitsky
✔ Parallel Consumption
✔ Batch Consumer
✔ Consumer Retry Strategies
✔ Resilient Producer
✔ Context Propagation
✔ Metrics reporting
Self Service 🟊
Greyhound
Kafka- based Global Data Mesh
Kafka Broker
@NSilnitsky
Scala NodeJS / Python / Go
Greyhound
Kafka Broker
I’m a container
gRPC + proto
sidecar
JVM library
Kafka- based Global Data Mesh
@NSilnitsky
Kafka Broker
Domain Events
ETL to DWH
Task Scheduling
Async Queue
...
We use it
for...
Kafka- based Global Data Mesh
Infra
@NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness
@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Kafka Broker
Single
Cluster
Overload
* intra-dc
@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Single
Cluster
Overload
A
A
A
A
B
Kafka Cluster (by SLAs)
B
Multi
Cluster
C
@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
A
A
A
A
B
To which
Cluster?
Kafka Cluster (by SLAs)
B
I want to produce
a domain event.
C
@NSilnitsky
@NSilnitsky
aGreyhoundConsumerSpec(groupName, messageHandler, topicName)
.ClusterA
val producer = GreyhoundBuilder.resilientProducer(topicName,
_.ClusterB
Kafka- based Global Data Mesh
* wix-layer
@NSilnitsky
@NSilnitsky
@NSilnitsky
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Kafka + Greyhound
Growing
Pains
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster
@NSilnitsky
@NSilnitsky
DC1 DC2
DC3 DC4
Master Active
Master Passive
Read-Only DBs
Http/gRPC
To which
DC?
I want to
process the event
and save the data.
@NSilnitsky
@NSilnitsky
Kafka + Greyhound
+ Cross DC services
Kafka- based Global Data Mesh
Growing
Pains
Hundreds of
developers
Event-driven
architecture
Diverse
tech stack
Large scale
traffic
Cluster
Overload
DC
awareness
+ Multi Cluster
@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka
@NSilnitsky
Replicate !
DC1 DC2
across DCs
Greyhound
producer
Greyhound
consumer
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
A
@NSilnitsky
Greyhound
Batch
Consumer
Greyhound
Producer
Batch #
Replicate !
Kafka
Broker
Cluster
A
Kafka
Broker
Cluster
DC1 DC2
How?
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh
A
Cross DC
Replication
Service
@NSilnitsky
Replicate !
DC1 DC2
a Compact Topic for Policy
A
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh
@NSilnitsky
Replication
service
Replicate !
DC1 DC2
How?
A
KVStoreReader
In-Memory Map
Key Value
Topic1 AllDCs
Topic2 Custom
A
Kafka
Broker
Cluster
Kafka
Broker
Cluster
Greyhound
producer
Greyhound
consumer
Kafka- based Global Data Mesh
@NSilnitsky
Kafka- based Global Data Mesh
@NSilnitsky
Kafka- based Global Data Mesh
@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka
@NSilnitsky
Produce !
DC1 DC2
To main DC
Read-Only DB /
Master Passive DB
Master Active
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer
@NSilnitsky
main-dc-producer
service
Produce !
DC1 DC2
How?
Master Active
Kafka- based Global Data Mesh
Greyhound
gRPC
producer
producer(topicName, _.toMainDc)
@NSilnitsky
@NSilnitsky
Cross-DC
Self Services 🟊
1. ✔ Replicate across DCs
2. ✔ Produce to Main DC
[WIP] Just Produce ❤
Greyhound
Kafka
@NSilnitsky
DC1 DC2
To where needed
[WIP] Just Produce ❤
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
Master Active
@NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Master Active
@NSilnitsky
Topology
Topic List
[consumer
group:DC]
DC1 DC2
A
Kafka
Broker
Cluster
Master Active
Kafka- based Global Data Mesh
Greyhound
producer
Greyhound
consumer
To where needed
[WIP] Just Produce ❤
Replication service
* reintroduce decoupling
@NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
DC1 DC2
DC3 DC4
A
A
A
A
B
B
Just Produce ❤
To a Single Logical
Kafka Cluster.
Kafka Broker
Cluster
DB
@NSilnitsky
@NSilnitsky
Kafka- based Global Data Mesh
Any
Tech Stack
Greyhound
Anywhere
You Need
developer
@NSilnitsky
developer
github.com/wix/greyhound
Kafka- based Global Data Mesh
@NSilnitsky
@NSilnitsky
Thank
You!
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
👉 slideshare.net/NatanSilnitsky
Any questions?
Kafka- based Global Data Mesh
@NSilnitsky
@NSilnitsky
Q&A
natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
👉 slideshare.net/NatanSilnitsky
Any questions?
Kafka- based Global Data Mesh

More Related Content

PDF
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
PDF
Jax london - Battle-tested event-driven patterns for your microservices archi...
PDF
Open sourcing a successful internal project - Reversim 2021
PDF
Advanced Caching Patterns used by 2000 microservices - Api World
PPTX
Standardizing Microservice Management With a Service Mesh
PDF
Battle-tested event-driven patterns for your microservices architecture - Sca...
PDF
5 lessons learned for Successful Migration to Confluent Cloud
PDF
Advanced Microservices Caching Patterns - Devoxx UK
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...
Jax london - Battle-tested event-driven patterns for your microservices archi...
Open sourcing a successful internal project - Reversim 2021
Advanced Caching Patterns used by 2000 microservices - Api World
Standardizing Microservice Management With a Service Mesh
Battle-tested event-driven patterns for your microservices architecture - Sca...
5 lessons learned for Successful Migration to Confluent Cloud
Advanced Microservices Caching Patterns - Devoxx UK

What's hot (20)

PDF
Advanced Caching Patterns used by 2000 microservices - Devoxx Ukraine
PDF
Greyhound - Powerful Pure Functional Kafka Library
PDF
Advanced Caching Patterns used by 2000 microservices - Code Motion
PPTX
Spring Cloud: API gateway upgrade & configuration in the cloud
PDF
Network Service Mesh
PDF
Kubernetes-Native DevOps: For Apache Kafka® with Confluent
PPTX
Communication Amongst Microservices: Kubernetes, Istio, and Spring Cloud - An...
PDF
Day in the life event-driven workshop
PDF
Using Microservices Architecture and Patterns to Address Applications Require...
PDF
The what, why and how of knative
PPTX
Cloud native microservices for systems and applications ieee rev2
PDF
Knative from an Enterprise Perspective
PDF
WTF Do We Need a Service Mesh?
PDF
Ambassador Kubernetes-Native API Gateway
PDF
Handling GDPR with Apache Kafka: How to Comply Without Freaking Out? (David J...
KEY
Modular Web Applications With Netzke
PDF
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
PDF
Building and Running Workloads the Knative Way
PDF
Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov ...
PPTX
Kubernetes + netflix oss
Advanced Caching Patterns used by 2000 microservices - Devoxx Ukraine
Greyhound - Powerful Pure Functional Kafka Library
Advanced Caching Patterns used by 2000 microservices - Code Motion
Spring Cloud: API gateway upgrade & configuration in the cloud
Network Service Mesh
Kubernetes-Native DevOps: For Apache Kafka® with Confluent
Communication Amongst Microservices: Kubernetes, Istio, and Spring Cloud - An...
Day in the life event-driven workshop
Using Microservices Architecture and Patterns to Address Applications Require...
The what, why and how of knative
Cloud native microservices for systems and applications ieee rev2
Knative from an Enterprise Perspective
WTF Do We Need a Service Mesh?
Ambassador Kubernetes-Native API Gateway
Handling GDPR with Apache Kafka: How to Comply Without Freaking Out? (David J...
Modular Web Applications With Netzke
Microservice 4.0 Journey - From Spring NetFlix OSS to Istio Service Mesh and ...
Building and Running Workloads the Knative Way
Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov ...
Kubernetes + netflix oss
Ad

Similar to Kafka based Global Data Mesh at Wix (20)

PDF
Kafka Summit London - Kafka based Global Data Mesh at Wix
PDF
Dev Days Europe - Kafka based Global Data Mesh at Wix
PDF
Kafka based Global Data Mesh at Wix with Natan Silnitsky | Kafka Summit Londo...
PDF
Devoxx Ukraine - Kafka based Global Data Mesh
PDF
Devoxx UK - Migrating to Multi Cluster Managed Kafka
PDF
Migrating to Multi Cluster Managed Kafka - ApacheKafkaIL
PDF
Introduction to apache kafka, confluent and why they matter
PDF
Migrating to Multi Cluster Managed Kafka - DevopStars 2022
PDF
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
PDF
5 lessons learned for successful migration to Confluent cloud | Natan Silnits...
PDF
Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative
PDF
Building Real-Time Travel Alerts
PDF
Kafka Vienna Meetup 020719
PPTX
Data Streaming with Apache Kafka & MongoDB - EMEA
PPTX
Webinar: Data Streaming with Apache Kafka & MongoDB
PDF
WSO2Con ASIA 2016: WSO2 Cloud Strategy Update
PDF
WSO2 Cloud Strategy Update
PDF
Battle Tested Event-Driven Patterns for your Microservices Architecture - Ris...
PDF
Battle Tested Event-Driven Patterns for your Microservices Architecture
PPTX
IoT and Event Streaming at Scale with Apache Kafka
Kafka Summit London - Kafka based Global Data Mesh at Wix
Dev Days Europe - Kafka based Global Data Mesh at Wix
Kafka based Global Data Mesh at Wix with Natan Silnitsky | Kafka Summit Londo...
Devoxx Ukraine - Kafka based Global Data Mesh
Devoxx UK - Migrating to Multi Cluster Managed Kafka
Migrating to Multi Cluster Managed Kafka - ApacheKafkaIL
Introduction to apache kafka, confluent and why they matter
Migrating to Multi Cluster Managed Kafka - DevopStars 2022
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
5 lessons learned for successful migration to Confluent cloud | Natan Silnits...
Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative
Building Real-Time Travel Alerts
Kafka Vienna Meetup 020719
Data Streaming with Apache Kafka & MongoDB - EMEA
Webinar: Data Streaming with Apache Kafka & MongoDB
WSO2Con ASIA 2016: WSO2 Cloud Strategy Update
WSO2 Cloud Strategy Update
Battle Tested Event-Driven Patterns for your Microservices Architecture - Ris...
Battle Tested Event-Driven Patterns for your Microservices Architecture
IoT and Event Streaming at Scale with Apache Kafka
Ad

More from Natan Silnitsky (17)

PDF
Async-ronizing Success at Wix - Patterns for Seamless Microservices - Devoxx ...
PDF
Integration Ignited Redefining Event-Driven Architecture at Wix - EventCentric
PDF
Reinventing Microservices Efficiency and Innovation with Single-Runtime
PDF
Async Excellence Unlocking Scalability with Kafka - Devoxx Greece
PDF
Wix Single-Runtime - Conquering the multi-service challenge
PDF
WeAreDevs - Supercharge Your Developer Journey with Tiny Atomic Habits
PDF
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
PDF
Effective Strategies for Wix's Scaling challenges - GeeCon
PDF
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
PDF
Workflow Engines & Event Streaming Brokers - Can they work together? [Current...
PDF
DevSum - Lessons Learned from 2000 microservices
PDF
GeeCon - Lessons Learned from 2000 microservices
PDF
Wix+Confluent Meetup - Lessons Learned from 2000 Event Driven Microservices
PDF
BuildStuff - Lessons Learned from 2000 Event Driven Microservices
PDF
Lessons Learned from 2000 Event Driven Microservices - Reversim
PDF
5 Takeaways from Migrating a Library to Scala 3 - Scala Love
PDF
How to successfully manage a ZIO fiber’s lifecycle - Functional Scala 2021
Async-ronizing Success at Wix - Patterns for Seamless Microservices - Devoxx ...
Integration Ignited Redefining Event-Driven Architecture at Wix - EventCentric
Reinventing Microservices Efficiency and Innovation with Single-Runtime
Async Excellence Unlocking Scalability with Kafka - Devoxx Greece
Wix Single-Runtime - Conquering the multi-service challenge
WeAreDevs - Supercharge Your Developer Journey with Tiny Atomic Habits
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
Effective Strategies for Wix's Scaling challenges - GeeCon
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Workflow Engines & Event Streaming Brokers - Can they work together? [Current...
DevSum - Lessons Learned from 2000 microservices
GeeCon - Lessons Learned from 2000 microservices
Wix+Confluent Meetup - Lessons Learned from 2000 Event Driven Microservices
BuildStuff - Lessons Learned from 2000 Event Driven Microservices
Lessons Learned from 2000 Event Driven Microservices - Reversim
5 Takeaways from Migrating a Library to Scala 3 - Scala Love
How to successfully manage a ZIO fiber’s lifecycle - Functional Scala 2021

Recently uploaded (20)

PPTX
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
PDF
Softaken Excel to vCard Converter Software.pdf
PDF
Understanding Forklifts - TECH EHS Solution
PDF
2025 Textile ERP Trends: SAP, Odoo & Oracle
PDF
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
PDF
wealthsignaloriginal-com-DS-text-... (1).pdf
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
PDF
AI in Product Development-omnex systems
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
PDF
Design an Analysis of Algorithms I-SECS-1021-03
PDF
Adobe Premiere Pro 2025 (v24.5.0.057) Crack free
PDF
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
PDF
Navsoft: AI-Powered Business Solutions & Custom Software Development
PDF
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
PPTX
Transform Your Business with a Software ERP System
PDF
System and Network Administration Chapter 2
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
Softaken Excel to vCard Converter Software.pdf
Understanding Forklifts - TECH EHS Solution
2025 Textile ERP Trends: SAP, Odoo & Oracle
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
wealthsignaloriginal-com-DS-text-... (1).pdf
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
AI in Product Development-omnex systems
Adobe Illustrator 28.6 Crack My Vision of Vector Design
Wondershare Filmora 15 Crack With Activation Key [2025
Design an Analysis of Algorithms I-SECS-1021-03
Adobe Premiere Pro 2025 (v24.5.0.057) Crack free
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
VVF-Customer-Presentation2025-Ver1.9.pptx
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
Navsoft: AI-Powered Business Solutions & Custom Software Development
Why TechBuilder is the Future of Pickup and Delivery App Development (1).pdf
Transform Your Business with a Software ERP System
System and Network Administration Chapter 2

Kafka based Global Data Mesh at Wix