SlideShare a Scribd company logo

Riding the Streaming Wave DIY style

Download as PPTX, PDF
K
Konstantine Karantasis

Kafka Connect allows developers to easily build plugins that integrate data from various sources and sinks. The document discusses how to develop Kafka Connect plugins using Confluent Open Source tools. It recommends using the Confluent CLI for local development and testing due to features like classloading isolation. Debugging plugins is also made simple by exporting environment variables and attaching a remote debugger. Once developed, plugins can be packaged and published for use in Kafka Connect.

Technology
Related topics:
Apache Kafka
1 of 28
Downloaded 12 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
1 Riding the Streaming Wave DIY style: Using & Building Kafka Connect Plugins with Confluent Open Source Konstantine Karantasis, Software Engineer
2 Intro Contributor: • Apache Kafka • Confluent Open Source • Confluent Enterprise
3 Challenge …
4 Challenge …
5 … and Answer Streaming data platform as the central nervous system of data architecture.
6 Kafka Connect: An Overview
7 Kafka Connect: An Overview
8 Kafka Connect: An Overview
9 Kafka Connect: An Overview
10 Kafka Connect: An Overview
11 Kafka Connect: An Overview
12 Kafka Connect: An Overview
13 Kafka Connect: Out-of-the-box Get for free: • Distributed deployment that scales. • Lean multi-tenancy with packaging/classloading isolation. • REST API to manage plugins. • Easy to use interfaces for automatic and periodic as well as manual tracking of progress.
14 Kafka Connect: Out-of-the-box more: • Native integration with monitoring platforms such as Confluent Control Center. • Metrics (more coming up soon). • Continuous open source development.
15 Kafka Connect Plugins: A Developer’s Perspective An ecosystem of plugins: • Connectors • Transforms • Converters How data flow through Connect API:
16 Kafka Connect Plugin Types Connectors are the richest plugins. Two types: • Source Connectors • Sink Connectors Connectors may support both structured and unstructured data: • Converters and the Schema Registry Transforms: • Is data useful as is? Or can use some basic transformations?
17 Where to start Sounds great! How do I start? (or else: Why Confluent Open Source for Connect plugin dev)
18 Where to start Iterate fast during development with Confluent CLI:
19 Classloading Isolation: Development with peace of mind Use the plugin.path worker configuration property my-plugins (included in the plugin.path ) kafka-connect-foo-connector JAR files, sample configs, licenses, etc.
20 Classloading Isolation: Development with peace of mind Workers isolate the JARs for each connector, transform, and converter to prevent conflicts. my-plugins (included in the plugin.path ) kafka-connect-foo-connector kafka-connect-bar-connector kafka-connect-baz-uber.jar
21 Kafka Connect in Action Let’s build a simple stream of data with Kafka Connect. • Find a dev-friendly public feed (e.g. meetup rsvps) • Start a simple source connector (here: file source connector) Demo with Confluent CLI
22 Source Connector Dev: Basic Concepts • Keep track of Source Connector’s progress: User-defined source offsets. • It’s a distributed system. Design for multiple workers/multiple tasks per worker. • Map your data to topic-partitions efficiently. • At-least-once semantics for Source Connectors.
23 Kafka Connect in Action • Next, start a sink connector to extend the stream (here: s3 sink connector) Demo with Confluent CLI
24 Sink Connector Dev: Basic Concepts • Sink Connectors utilize Kafka consumer offsets to track progress. • By default at-least once semantics with automatic and periodic offset commits. • But if the Sink allows for determinism and idempotence, sink connectors can be exactly-once! • Examples: S3 and HDFS Connectors.
25 Debugging Kafka Connect and Connect Plugins This looked nice, but also pretty ideal for Connector development. How do I debug my Connect plugins? • Set the right environment variables for Confluent CLI export CONNECT_DEBUG=y; export DEBUG_SUSPEND_FLAG=y; • also: export CONFLUENT_CURRENT=/your/fav/dir/location (doesn’t have to be have to be /tmp) • Restart Kafka Connect worker • Attach a remote debugger using your favorite IDE
26 Package and Publish your connector So you built your Kafka Connect plugin! • Currently follow a commonly used pattern using maven- assembly-plugin or shade plugin • maven-kafka-connect-plugin coming up soon. • confluent.io/connectors
27 Summary Why develop with Kafka Connect? • Lot’s of functionality for free (scalability, multi-tenancy, mgmt, monitoring) • Quick on-boarding • Active community How to develop your Kafka Connect plugins? • Use Confluent CLI • Use classloading isolation with plugin.path • Debug your connector with your favorite IDE • Extend existing open source plugins or build your own! • Stay tuned, the best is yet to come!
28 Thank You!

More Related Content

PPTX
Analytics Beyond RAM Capacity using R
Alex Palamides
 
PDF
Athens BigData Meetup - Sept 17
Landoop Ltd
 
PPTX
Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017
Michael Noll
 
PDF
Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)
Keigo Suda
 
PPTX
Confluent building a real-time streaming platform using kafka streams and k...
Thomas Alex
 
PDF
Data Pipelines Made Simple with Apache Kafka
confluent
 
PDF
Kafka Summit SF 2017 - Database Streaming at WePay
confluent
 
PDF
Apache Kafka lessons learned @PAYBACK
Maxim Shelest
 
Analytics Beyond RAM Capacity using R
Alex Palamides
 
Athens BigData Meetup - Sept 17
Landoop Ltd
 
Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017
Michael Noll
 
Apache Kafka & Kafka Connectを に使ったデータ連携パターン(改めETLの実装)
Keigo Suda
 
Confluent building a real-time streaming platform using kafka streams and k...
Thomas Alex
 
Data Pipelines Made Simple with Apache Kafka
confluent
 
Kafka Summit SF 2017 - Database Streaming at WePay
confluent
 
Apache Kafka lessons learned @PAYBACK
Maxim Shelest
 

What's hot (20)

PPTX
Kafka Summit NYC 2017 Hanging Out with Your Past Self in VR
confluent
 
PPTX
Portable Streaming Pipelines with Apache Beam
confluent
 
PPTX
Taking a look under the hood of Apache Flink's relational APIs.
Fabian Hueske
 
PDF
Real-world Streaming Architectures
confluent
 
PDF
How to use Standard SQL over Kafka: From the basics to advanced use cases | F...
HostedbyConfluent
 
PDF
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
HostedbyConfluent
 
PDF
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
confluent
 
PDF
Data integration with Apache Kafka
confluent
 
PDF
Intro to AsyncAPI
confluent
 
PDF
Monitoring Apache Kafka with Confluent Control Center
confluent
 
PDF
Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...
HostedbyConfluent
 
PDF
Apache kafka-a distributed streaming platform
confluent
 
PDF
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
HostedbyConfluent
 
PDF
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
confluent
 
PDF
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Bowen Li
 
PDF
Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...
Michael Noll
 
PDF
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
Flink Forward
 
PDF
Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...
confluent
 
PPTX
2018 07-11 - kafka integration patterns
Alberto Paro
 
PPTX
Flink Forward Berlin 2017: Till Rohrmann - From Apache Flink 1.3 to 1.4
Flink Forward
 
Kafka Summit NYC 2017 Hanging Out with Your Past Self in VR
confluent
 
Portable Streaming Pipelines with Apache Beam
confluent
 
Taking a look under the hood of Apache Flink's relational APIs.
Fabian Hueske
 
Real-world Streaming Architectures
confluent
 
How to use Standard SQL over Kafka: From the basics to advanced use cases | F...
HostedbyConfluent
 
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
HostedbyConfluent
 
Creating Connector to Bridge the Worlds of Kafka and gRPC at Wework (Anoop Di...
confluent
 
Data integration with Apache Kafka
confluent
 
Intro to AsyncAPI
confluent
 
Monitoring Apache Kafka with Confluent Control Center
confluent
 
Failing to Cross the Streams – Lessons Learned the Hard Way | Philip Schmitt,...
HostedbyConfluent
 
Apache kafka-a distributed streaming platform
confluent
 
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
HostedbyConfluent
 
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
confluent
 
Apache Flink @ Alibaba - Seattle Apache Flink Meetup
Bowen Li
 
Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...
Michael Noll
 
Vyacheslav Zholudev – Flink, a Convenient Abstraction Layer for Yarn?
Flink Forward
 
Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...
confluent
 
2018 07-11 - kafka integration patterns
Alberto Paro
 
Flink Forward Berlin 2017: Till Rohrmann - From Apache Flink 1.3 to 1.4
Flink Forward
 
Ad

Similar to Riding the Streaming Wave DIY style (20)

PPTX
Data Pipelines with Kafka Connect
Kaufman Ng
 
PDF
Integrating Apache Kafka Into Your Environment
confluent
 
PPT
Kafka Explainaton
NguyenChiHoangMinh
 
PDF
Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Field
confluent
 
PPTX
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Data Con LA
 
PDF
Building Streaming Data Applications Using Apache Kafka
Slim Baltagi
 
PDF
Introduction to Kafka Streams - Knolx.pdf
Knoldus Inc.
 
PPTX
Kafka Streams for Java enthusiasts
Slim Baltagi
 
PPTX
kafka for db as postgres
PivotalOpenSourceHub
 
PDF
Introducing Kafka's Streams API
confluent
 
PPTX
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OS
Lightbend
 
PPTX
Being Ready for Apache Kafka - Apache: Big Data Europe 2015
Michael Noll
 
PDF
Deploying Kafka on DC/OS
Kaufman Ng
 
PDF
Day in the life event-driven workshop
Christina Lin
 
PDF
Apache Kafka - Scalable Message-Processing and more !
Guido Schmutz
 
ODP
Introduction to Kafka connect
Knoldus Inc.
 
PPTX
Kafka connect 101
Whiteklay
 
PDF
Building Microservices with Apache Kafka
confluent
 
PDF
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Kai Wähner
 
PDF
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
confluent
 
Data Pipelines with Kafka Connect
Kaufman Ng
 
Integrating Apache Kafka Into Your Environment
confluent
 
Kafka Explainaton
NguyenChiHoangMinh
 
Kafka Summit SF 2017 - Kafka Connect Best Practices – Advice from the Field
confluent
 
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Data Con LA
 
Building Streaming Data Applications Using Apache Kafka
Slim Baltagi
 
Introduction to Kafka Streams - Knolx.pdf
Knoldus Inc.
 
Kafka Streams for Java enthusiasts
Slim Baltagi
 
kafka for db as postgres
PivotalOpenSourceHub
 
Introducing Kafka's Streams API
confluent
 
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OS
Lightbend
 
Being Ready for Apache Kafka - Apache: Big Data Europe 2015
Michael Noll
 
Deploying Kafka on DC/OS
Kaufman Ng
 
Day in the life event-driven workshop
Christina Lin
 
Apache Kafka - Scalable Message-Processing and more !
Guido Schmutz
 
Introduction to Kafka connect
Knoldus Inc.
 
Kafka connect 101
Whiteklay
 
Building Microservices with Apache Kafka
confluent
 
Event-Driven Stream Processing and Model Deployment with Apache Kafka, Kafka ...
Kai Wähner
 
Event-Driven Model Serving: Stream Processing vs. RPC with Kafka and TensorFl...
confluent
 
Ad

Recently uploaded (20)

PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
KodekX | Application Modernization Development
KodekX
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Cloud computing and distributed systems.
kkkkkhan
 
KodekX | Application Modernization Development
KodekX
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Encapsulation theory and applications.pdf
gurumoop
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 

Riding the Streaming Wave DIY style

  • 1. 1 Riding the Streaming Wave DIY style: Using & Building Kafka Connect Plugins with Confluent Open Source Konstantine Karantasis, Software Engineer
  • 2. 2 Intro Contributor: • Apache Kafka • Confluent Open Source • Confluent Enterprise
  • 5. 5 … and Answer Streaming data platform as the central nervous system of data architecture.
  • 13. 13 Kafka Connect: Out-of-the-box Get for free: • Distributed deployment that scales. • Lean multi-tenancy with packaging/classloading isolation. • REST API to manage plugins. • Easy to use interfaces for automatic and periodic as well as manual tracking of progress.
  • 14. 14 Kafka Connect: Out-of-the-box more: • Native integration with monitoring platforms such as Confluent Control Center. • Metrics (more coming up soon). • Continuous open source development.
  • 15. 15 Kafka Connect Plugins: A Developer’s Perspective An ecosystem of plugins: • Connectors • Transforms • Converters How data flow through Connect API:
  • 16. 16 Kafka Connect Plugin Types Connectors are the richest plugins. Two types: • Source Connectors • Sink Connectors Connectors may support both structured and unstructured data: • Converters and the Schema Registry Transforms: • Is data useful as is? Or can use some basic transformations?
  • 17. 17 Where to start Sounds great! How do I start? (or else: Why Confluent Open Source for Connect plugin dev)
  • 18. 18 Where to start Iterate fast during development with Confluent CLI:
  • 19. 19 Classloading Isolation: Development with peace of mind Use the plugin.path worker configuration property my-plugins (included in the plugin.path ) kafka-connect-foo-connector JAR files, sample configs, licenses, etc.
  • 20. 20 Classloading Isolation: Development with peace of mind Workers isolate the JARs for each connector, transform, and converter to prevent conflicts. my-plugins (included in the plugin.path ) kafka-connect-foo-connector kafka-connect-bar-connector kafka-connect-baz-uber.jar
  • 21. 21 Kafka Connect in Action Let’s build a simple stream of data with Kafka Connect. • Find a dev-friendly public feed (e.g. meetup rsvps) • Start a simple source connector (here: file source connector) Demo with Confluent CLI
  • 22. 22 Source Connector Dev: Basic Concepts • Keep track of Source Connector’s progress: User-defined source offsets. • It’s a distributed system. Design for multiple workers/multiple tasks per worker. • Map your data to topic-partitions efficiently. • At-least-once semantics for Source Connectors.
  • 23. 23 Kafka Connect in Action • Next, start a sink connector to extend the stream (here: s3 sink connector) Demo with Confluent CLI
  • 24. 24 Sink Connector Dev: Basic Concepts • Sink Connectors utilize Kafka consumer offsets to track progress. • By default at-least once semantics with automatic and periodic offset commits. • But if the Sink allows for determinism and idempotence, sink connectors can be exactly-once! • Examples: S3 and HDFS Connectors.
  • 25. 25 Debugging Kafka Connect and Connect Plugins This looked nice, but also pretty ideal for Connector development. How do I debug my Connect plugins? • Set the right environment variables for Confluent CLI export CONNECT_DEBUG=y; export DEBUG_SUSPEND_FLAG=y; • also: export CONFLUENT_CURRENT=/your/fav/dir/location (doesn’t have to be have to be /tmp) • Restart Kafka Connect worker • Attach a remote debugger using your favorite IDE
  • 26. 26 Package and Publish your connector So you built your Kafka Connect plugin! • Currently follow a commonly used pattern using maven- assembly-plugin or shade plugin • maven-kafka-connect-plugin coming up soon. • confluent.io/connectors
  • 27. 27 Summary Why develop with Kafka Connect? • Lot’s of functionality for free (scalability, multi-tenancy, mgmt, monitoring) • Quick on-boarding • Active community How to develop your Kafka Connect plugins? • Use Confluent CLI • Use classloading isolation with plugin.path • Debug your connector with your favorite IDE • Extend existing open source plugins or build your own! • Stay tuned, the best is yet to come!