SlideShare a Scribd company logo
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Building a Data Subscription
Service with Kafka Connect
Kafka Summit 2020
24 August 2020
Danica Fine, Software Engineer
Ajay Vyasapeetam, Software Engineer
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Use Case
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Non-Ideal Architecture
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Ideal Architecture
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Motivation
• Easy subscription to multiple data sources
• Consistent interfaces
§ Subscribing
§ Receiving data
• Optimize reuse across applications
• Fault-tolerant and scalable
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
System Architecture
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Subscription Manager
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Side Input Data Connector
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Building a Custom
Source Connector
© 2018 Bloomberg Finance L.P. All rights reserved.
Kafka Connect Basics
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Connect Components
• Connector
• Tasks
• Monitor Thread
© 2018 Bloomberg Finance L.P. All rights reserved.
Source Connector
• Defines task breakdown
• Kicks off monitor thread
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Source Connector Task
• Defines:
§ Where to collect data
§ How to format it
• Client for external data call
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Monitoring Thread
• Consumes from Kafka topics
• Triggers task reconfiguration
© 2018 Bloomberg Finance L.P. All rights reserved.
Final Workflow
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Pitfalls
• Kafka Connect stop-the-world reconfiguration
• Consumers in monitor thread vs tasks
• Testing connectors
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
Questions?
Danica Fine, dfine17@bloomberg.net
Ajay Vyasapeetam, avyasapeeta1@bloomberg.net
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2020 Bloomberg Finance L.P. All rights reserved.
We are hiring!
Thank you!
https://www.bloomberg.com/careers

More Related Content

PDF
Fully-Managed, Multi-Tenant Kafka Clusters: Tips, Tricks, and Tools (Christop...
PDF
Help, My Kafka is Broken! (Emma Humber & Gantigmaa Selenge, IBM) Kafka Summit...
PPTX
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
PDF
Real-Time Market Data Analytics Using Kafka Streams
PDF
Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Blo...
PPTX
Data governance and discoverability at AO.com | Jon Vines, AO.com and Christo...
PDF
A Solution for Leveraging Kafka to Provide End-to-End ACID Transactions
PDF
HOP! Airlines Jets to Real Time
Fully-Managed, Multi-Tenant Kafka Clusters: Tips, Tricks, and Tools (Christop...
Help, My Kafka is Broken! (Emma Humber & Gantigmaa Selenge, IBM) Kafka Summit...
Building a Codeless Log Pipeline w/ Confluent Sink Connector | Pollyanna Vale...
Real-Time Market Data Analytics Using Kafka Streams
Replaying KStreams Apps Using State Snapshots (Nishchay Sinha & Yan Wang, Blo...
Data governance and discoverability at AO.com | Jon Vines, AO.com and Christo...
A Solution for Leveraging Kafka to Provide End-to-End ACID Transactions
HOP! Airlines Jets to Real Time

What's hot (20)

PPTX
Kafka Connect and KSQL: Useful Tools in Migrating from a Legacy System to Kaf...
PDF
Building Stateful applications on Streaming Platforms | Premjit Mishra, Dell ...
PDF
Cisco’s E-Commerce Transformation Using Kafka
PDF
Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...
PPTX
Apache Kafka Architectures and Fundamentals
PPTX
Comparing three data ingestion approaches where Apache Kafka integrates with ...
PDF
Death of the dumb pipes: Using Apache Kafka® for Integration projects
PDF
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
PPTX
Should we manage events like APIs? | Kim Clark, IBM
PDF
Elastically Scaling Kafka Using Confluent
PDF
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, Solace
PDF
Application Modernization Using Event Streaming Architecture (David Wadden, V...
PDF
Government Track Welcome Address
PDF
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
PPTX
Using Kafka Streams to Analyze Live Trading Activity for Crypto Exchanges (Lu...
PDF
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
PDF
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
PDF
Apache Kafka and the Data Mesh | Michael Noll, Confluent
PPTX
Confluent Private Cloud | Rohit Bakhshi, Staff Product Manager
PDF
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Kafka Connect and KSQL: Useful Tools in Migrating from a Legacy System to Kaf...
Building Stateful applications on Streaming Platforms | Premjit Mishra, Dell ...
Cisco’s E-Commerce Transformation Using Kafka
Kafka for connected vehicle research | Pavle Bujanovic, Federal Highway Admin...
Apache Kafka Architectures and Fundamentals
Comparing three data ingestion approaches where Apache Kafka integrates with ...
Death of the dumb pipes: Using Apache Kafka® for Integration projects
Transform Your Mainframe and IBM i Data for the Cloud with Precisely and Apac...
Should we manage events like APIs? | Kim Clark, IBM
Elastically Scaling Kafka Using Confluent
Mesh-ing around with Streams across the Enterprise | Phil Scanlon, Solace
Application Modernization Using Event Streaming Architecture (David Wadden, V...
Government Track Welcome Address
Why Kafka Works the Way It Does (And Not Some Other Way) | Tim Berglund, Conf...
Using Kafka Streams to Analyze Live Trading Activity for Crypto Exchanges (Lu...
Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...
Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...
Apache Kafka and the Data Mesh | Michael Noll, Confluent
Confluent Private Cloud | Rohit Bakhshi, Staff Product Manager
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Ad

Similar to Building a Data Subscription Service with Kafka Connect (Danica Fine & Ajay Vyasapeetam, Bloomberg) Kafka Summit 2020 (20)

PDF
Verifying Apache Kafka-Based Data Pipelines With Subhangi Agarwala | Current ...
PDF
Multi-Tenant HBase Cluster - HBaseCon2018-final
PPTX
Dataworks | 2018-06-20 | Gimel data platform
PPTX
Gimel at Dataworks Summit San Jose 2018
PDF
QCon 2018 | Gimel | PayPal's Analytic Platform
PPTX
apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...
PPTX
The Truth About the Service Mesh Data Plane
PPTX
The Real Costs of Bank Connectivity during SAP S/4HANA Migrations
PDF
The Power of Event Driven Caches (Brendan Powers, Bloomberg L.P) Kafka Summit...
PPTX
Next Generation of Treasury Technology Cash Adventure- Brad Teaver.pptx
PPTX
MCP Dev Summit - Pragmatic Scaling of Enterprise GenAI with MCP
PDF
Big Data LDN 2018: STREAM PROCESSING TAKES ON EVERYTHING
PDF
Kubernetes, Istio and Knative - noteworthy practical experience
PDF
Serving queries at low latency using HBase
PDF
HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies
PDF
Z api overview
PDF
2019 03-13-implementing microservices by ddd
PPTX
How data modelling helps serve billions of queries in millisecond latency wit...
PDF
Building a Streaming Microservices Architecture - Data + AI Summit EU 2020
PDF
Graph Gurus Episode 25: Unleash the Business Value of Your Data Lake with Gra...
Verifying Apache Kafka-Based Data Pipelines With Subhangi Agarwala | Current ...
Multi-Tenant HBase Cluster - HBaseCon2018-final
Dataworks | 2018-06-20 | Gimel data platform
Gimel at Dataworks Summit San Jose 2018
QCon 2018 | Gimel | PayPal's Analytic Platform
apidays LIVE New York 2021 - Simplify Open Policy Agent with Styra DAS by Tim...
The Truth About the Service Mesh Data Plane
The Real Costs of Bank Connectivity during SAP S/4HANA Migrations
The Power of Event Driven Caches (Brendan Powers, Bloomberg L.P) Kafka Summit...
Next Generation of Treasury Technology Cash Adventure- Brad Teaver.pptx
MCP Dev Summit - Pragmatic Scaling of Enterprise GenAI with MCP
Big Data LDN 2018: STREAM PROCESSING TAKES ON EVERYTHING
Kubernetes, Istio and Knative - noteworthy practical experience
Serving queries at low latency using HBase
HBaseConAsia2018 Track3-1: Serving billions of queries in millisecond latencies
Z api overview
2019 03-13-implementing microservices by ddd
How data modelling helps serve billions of queries in millisecond latency wit...
Building a Streaming Microservices Architecture - Data + AI Summit EU 2020
Graph Gurus Episode 25: Unleash the Business Value of Your Data Lake with Gra...
Ad

More from confluent (20)

PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
PDF
Migration, backup and restore made easy using Kannika
PDF
Five Things You Need to Know About Data Streaming in 2025
PDF
Data in Motion Tour Seoul 2024 - Keynote
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
PDF
Unlocking value with event-driven architecture by Confluent
PDF
Il Data Streaming per un’AI real-time di nuova generazione
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
PDF
Building API data products on top of your real-time data infrastructure
PDF
Speed Wins: From Kafka to APIs in Minutes
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
Webinar Think Right - Shift Left - 19-03-2025.pptx
Migration, backup and restore made easy using Kannika
Five Things You Need to Know About Data Streaming in 2025
Data in Motion Tour Seoul 2024 - Keynote
Data in Motion Tour Seoul 2024 - Roadmap Demo
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
Data in Motion Tour 2024 Riyadh, Saudi Arabia
Build a Real-Time Decision Support Application for Financial Market Traders w...
Strumenti e Strategie di Stream Governance con Confluent Platform
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
Building Real-Time Gen AI Applications with SingleStore and Confluent
Unlocking value with event-driven architecture by Confluent
Il Data Streaming per un’AI real-time di nuova generazione
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Break data silos with real-time connectivity using Confluent Cloud Connectors
Building API data products on top of your real-time data infrastructure
Speed Wins: From Kafka to APIs in Minutes
Evolving Data Governance for the Real-time Streaming and AI Era

Recently uploaded (20)

PDF
Unlocking AI with Model Context Protocol (MCP)
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PDF
Approach and Philosophy of On baking technology
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
PDF
Encapsulation theory and applications.pdf
PDF
cuic standard and advanced reporting.pdf
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
PPTX
Cloud computing and distributed systems.
PPT
Teaching material agriculture food technology
DOCX
The AUB Centre for AI in Media Proposal.docx
Unlocking AI with Model Context Protocol (MCP)
Dropbox Q2 2025 Financial Results & Investor Presentation
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
The Rise and Fall of 3GPP – Time for a Sabbatical?
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Approach and Philosophy of On baking technology
Digital-Transformation-Roadmap-for-Companies.pptx
Encapsulation theory and applications.pdf
cuic standard and advanced reporting.pdf
Understanding_Digital_Forensics_Presentation.pptx
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Building Integrated photovoltaic BIPV_UPV.pdf
Cloud computing and distributed systems.
Teaching material agriculture food technology
The AUB Centre for AI in Media Proposal.docx

Building a Data Subscription Service with Kafka Connect (Danica Fine & Ajay Vyasapeetam, Bloomberg) Kafka Summit 2020

  • 1. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Building a Data Subscription Service with Kafka Connect Kafka Summit 2020 24 August 2020 Danica Fine, Software Engineer Ajay Vyasapeetam, Software Engineer
  • 2. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Use Case
  • 3. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Non-Ideal Architecture
  • 4. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Ideal Architecture
  • 5. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Motivation • Easy subscription to multiple data sources • Consistent interfaces § Subscribing § Receiving data • Optimize reuse across applications • Fault-tolerant and scalable
  • 6. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. System Architecture
  • 7. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Subscription Manager
  • 8. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Side Input Data Connector
  • 9. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Building a Custom Source Connector
  • 10. © 2018 Bloomberg Finance L.P. All rights reserved. Kafka Connect Basics
  • 11. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Connect Components • Connector • Tasks • Monitor Thread
  • 12. © 2018 Bloomberg Finance L.P. All rights reserved. Source Connector • Defines task breakdown • Kicks off monitor thread
  • 13. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Source Connector Task • Defines: § Where to collect data § How to format it • Client for external data call
  • 14. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Monitoring Thread • Consumes from Kafka topics • Triggers task reconfiguration
  • 15. © 2018 Bloomberg Finance L.P. All rights reserved. Final Workflow
  • 16. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Pitfalls • Kafka Connect stop-the-world reconfiguration • Consumers in monitor thread vs tasks • Testing connectors
  • 17. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. Questions? Danica Fine, dfine17@bloomberg.net Ajay Vyasapeetam, avyasapeeta1@bloomberg.net
  • 18. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. © 2020 Bloomberg Finance L.P. All rights reserved. We are hiring! Thank you! https://www.bloomberg.com/careers