Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft
0 likes369 views
The document discusses Kafka connectors for Cosmos DB that allow for seamless integration between the two services without requiring complex application code. It provides an overview of Kafka Connect and connectors, use cases for integrating Cosmos DB and Kafka, and the architecture of source and sink connectors that can read from and write to Cosmos DB and Kafka. It also previews a demo of the connectors and suggests ways to take integration further.
Azure Cosmos DB Kafka Connectors | Abinav Rameesh, Microsoft
- 1. Cosmos DB – Kafka Connectors Abinav Rameesh Program Manager, Cosmos DB
- 2. 01 Kafka Connect Overview 02 Kafka Integration Use Cases for Cosmos DB 03 Cosmos DB Source & Sink Architecture Overview 04 Demo 05 Taking It Further
- 3. What is a Connector? Confluent Platform offers 120+ pre-built connectors to help you quickly and reliably integrate with Apache Kafka®. Connectors import and export data from some of the most commonly used data systems. Connectors either run as a managed resource on Confluent Cloud or as a self managed resource on a self managed kafka cluster. Kafka Connect runs under the Java virtual machine (JVM) as a process known as a worker. Each worker can execute multiple connectors.
- 4. Connect Architecture •Connectors are responsible for the interaction between Kafka Connect and the external technology being integrated with •Converters handle the serialization and deserialization of data •Transformations can optionally apply one or more transformations to the data passing through the pipeline
- 5. Kafka Cosmos DB SHARDING SCALING CDC TUNABILITY MANAGED Kafka is horizontally partitioned with brokers serving as leaders and followers, each owning it’s own logical range of data. Kafka can be scaled seamlessly by simply increasing the number of brokers in the cluster. Kafka provides Consumer library to retrieve changes from each of the physical partitions of the topic. Kafka performance can be tuned for batch sizes, memory thresholds, polling frequencies etc. Kafka can be self provisioned or fully managed through Confluent Cloud. Cosmos DB is horizontally partitioned with partitions composing a replica set. Cosmos DB can be scaled elastically by simply increasing the throughput for dataset. Cosmos DB provides a Change Feed Processor, which contains inbuilt logic to retrieve changes from the physical partitions for the container. Cosmos DB can be tuned for query performance, RU consumption, batch sizes for writes and reads etc. Cosmos DB is a fully managed service.
- 6. 01 Source and Sink Connectors for Cosmos DB facilitate seamless integration without the need to write complex application code to migrate data to and from Kafka. SOURCE, SINK ZERO CODE DATA FORMATS FULLY MANAGED 02 Only configurations are needed to point to the Cosmos DB account and Kafka cluster with additional customization options. 03 JSON and AVRO data formats are supported with additional format options to come based on user feedback. 04 Fully managed (through Confluent Cloud) as well as self managed (using the connector directly) are available.
- 7. Cosmos DB – Kafka Use Cases
- 9. Cosmos DB – Kafka Connector Architecture …. …. Managed Kafka Connect Cluster (Source Connector) Change Feed Processor Kafka Producer Reading from Cosmos DB’s physical partitions Writing to the Kafka topic’s physical partitions
- 10. Cosmos DB – Kafka Connector Architecture …. Managed Kafka Connect Cluster (Sink Connector) Cosmos DB Java Client issuing writes to the Cosmos container Kafka Consumer pulling from the topic’s brokers Writing to Cosmos DB’s physical partitions ….
- 11. Demo