How to Write Great Kafka Connectors
0 likes375 views
The document provides a comprehensive guide on writing effective Kafka connectors, outlining key elements such as connector classes, task management, configuration, error handling, and resource management. It emphasizes the importance of scalability, procedural integrity, and the proper management of offsets and delivery semantics. The guide also includes specific implementation tips for various external systems, cloud integration, and testing strategies to ensure performance and reliability.
How to Write Great Kafka Connectors
- 1. How to Write Great Connectors
- 2. 2 HashedIn Technologies A software development company that builds deep tech SAAS products and platforms. Experts in Analytics, Machine Learning & Integrations, building upon learnings and frameworks from a decade of launching 200+ products. hashedin.com
- 3. 3 HashedIn Technologies Speaker: Fahad Sheikh, Tech Lead, HashedIn He delivers Kafka Connectors and supports existing connectors. Q/A: Jeff Bean Partner Solution Architect, Confluent Supports Confluent partners through the Verified Integrations Program.
- 4. How to write great Kafka Connectors Fahad Sheikh
- 6. Exchange of Data 1. A typical data exchange between external systems and Apache Kafka 2. High coupling 3. Challenges a. How do I restart? b. How to scale the application? c. How to capture metrics? d. How to handle failures and retries? e. How to properly configure and use Producer and Consumer APIs
- 7. Exchange of Data with Kafka Connect 1. Offset Management 2. Parallelization 3. Task distribution 4. Configuration driven 5. Scalability 6. Metrics 7. REST APIs 8. Failure and retries 9. Schemas and data types
- 8. Kafka Connect 1. Simplifies and automates the integration with external systems 2. Loose coupling between systems 3. Reliable 4. Scalable 5. Connectors, SMTs, Converters …
- 11. Basic of Connector Programming 1. Connector Class a. Source Connector b. Sink Connector 2. Configuration Class a. Source Config b. Sink Config 3. Task Class a. Source Task b. Sink Task
- 12. Connector class
- 13. SourceTask class
- 14. SinkTask class
- 15. Considerations 1. Making Connector configurable 2. Task Distribution 3. Offset Management 4. Retries, Exception Handling and Logging 5. Schema and Data types 6. Delivery Semantics 7. Resource Management 8. Improving throughput 9. Security(Proxy and secure channels) 10. System Testing, Verification and Performance tests 11. Backwards Compatibility 12. Documentation
- 16. Make Connector configurable 1. Configuration driven 2. Validators and Recommenders. 3. Fail fast 4. Precise documentation 5. Make use of TYPE.Password
- 17. Task Distribution 1. Parallelism 2. Delegating work to the Connector tasks 3. Ideally, number of topic-partitions = number of Connector tasks 4. Task reconfiguration to address changes in external system
- 18. Offset Management 1. Source Connector a. Resume from the previous offset // OffsetStorageReader b. Make use of commitRecord() and commit() methods 1. Sink Connector a. Maintaining offsets // Buffering of records b. preCommit() // Tracking the actual records written to ext- system c. Committing offsets in external system
- 19. Controlling Delivery Semantics 1. Strive for Exactly once delivery a. No duplicates b. No data/records missed 2. Committing offsets frequently and accurately // Writing them to ext-system 3. Handling restarts and rebalances // Resuming from the last position 4. Deduplication, if required // While sourcing records
- 20. Retries, Exception Handling and Logging 1. Retrying the transient errors a. Constant/Exponential Backoff b. Configuration - max.retry.time.ms 2. Or, let Kafka Connect handle retries(throw RetriableException) 3. Throw ConfigException or ConnectException // Task in non-recoverable state 4. Fail fast // Have pre-conditional checks 5. Make sure not to swallow the exception // retain the trace 6. Use appropriate logging levels 7. Not logging sensitive data 8. Make behavior on error configurable(fail/log/ignore) //dead letter queue
- 21. Schema and Data types 1. Supported formats : Avro, JSON, String, Protobuf, …. 2. Preserve the schema, use Avro a. Extensible schema language defined in pure JSON b. Best notion of compatibility for evolving your data over time. c. Compact format. The bulk of JSON, repeating every field name with every single record, is what makes JSON inefficient for high-volume usage. d. Direct mapping to and from JSON e. Easy to build complex schemas // nesting is also feasible :) 1. Robustness 2. Clarity and Semantics 3. Compatibility 4. Eliminates the manual labor required for Data Science
- 22. Resource Management 1. Sharing of resource between tasks // thread safety 2. Keep a check on resources being used 3. Closing resources / Clean up // streams, connection objects, …. a. Operation level b. Task level c. Connector level 4. Pre-conditional check on Connector start to avoid occupying unnecessary resources, fail-fast!
- 23. Improving throughput 1. Multi task support 2. Batching operations 3. Reducing unnecessary network calls(applicable while committing records as well) a. One time login to the system b. Reusing connection objects c. ….
- 24. Security 1. Basic authentication 2. Proxy authentication 3. Support for SSL/TLS 4. NTLM / Kerberos authentication 5. Implement CredentialsProvider class // Example : for AWS services 6. Avoid writing sensitive data into Kafka topics
- 25. Unit testing, System testing and verification 1. Extensive unit tests a. Tests for all possible scenarios/inputs b. To expose any breaking changes 2. Integration tests 3. Testing in production like environment a. On-premise b. Cloud 4. Confluent verification guide 5. Be explicit with the version of the system supported in documentation a. Try supporting the oldest possible version
- 26. Performance testing 1. Run performance tests on production like environments 2. Run them for multiple tasks count 3. Make sure the environment doesn’t limit the performance 4. Check for memory leaks (Heap usage)
- 27. Documentation 1. Precise and detailed documentation 2. Explicitly mention versions supported 3. Limitations 4. Connector installation section 5. User guide section to walk users through the prerequisites and setup 6. Document each configuration 7. Document record schemas 8. Refer confluent connector documentation
- 28. Few tips for various External Systems 1. Connector implementation in case of: a. Pull based systems b. Pushed based systems 2. Databases 3. Messaging Queues 4. Incident/Workflow Management Systems 5. Data Lakes and Storage Systems 6. Monitoring Systems 7. Software development tools 8. General implementation related tips
- 29. Pull based systems 1. Source Connector implementation a) Acknowledge records/messages received and processed b) Appropriately configure the external system 2. Sink Connector implementation a) A little complicated implementation b) Writing records to buffer and flushing them to the external system when being pulled.
- 30. Push based systems 1. Source Connector implementation a) Start a separate thread and implement Listeners, if required. b) Maintain shared buffer between thread and task. c) Return records in stored in queue/buffer to Kafka Connect 2. Sink Connector implementation a) Possibly, batch records and write them to external system. b) Response logging
- 31. Databases 1. Refer JDBC Connector (https://github.com/confluentinc/kafka-connect-jdbc) 2. Data mapping 3. Key handling // pk.mode 4. Delete Mode 5. Idempotent writes // upsert mode 6. Auto Table creation 7. Schema evolution 8. Write offsets to database table
- 32. Messaging Queues 1. Configuring the queues correctly 2. Acknowledging the records received and processed 3. Multi-task support, if feasible 4. Retaining important data from the message a. Message topic/channel -> Kafka Header b. Message Id -> KafkaKey c. Message Body -> KafkaValue d. Message Headers(metadata) -> KafkaHeaders
- 33. Incident Management Systems 1. Important to achieve exactly-once-semantics 2. Retries in case of failures 3. Appropriately logging the response a. Connect logs b. To some other external system(may be a kafka topic) i. Logging success request ii. Logging failed requests with the Kafka coordinates of records
- 34. Data Lakes, Warehouses and Storage Systems 1. Multi task support and task distribution 2. Partitioners(fields based, time based, ….) 3. Support for various data formats (Avro, String JSON, ByteArray, Parquet,...) 4. Object uploads a. Based on records count b. Based on record time/rotation 5. Avoid re-processing of data // maintain offsets
- 35. Monitoring Systems 1. Support for different types of metrics 2. Try adhering to standard metrics structure as input // OpenCensus, …. 3. Retrying in case of failures 4. Batching the records 5. Regularly committing offsets to avoid corrupting the data in the system
- 36. Software Development tools 1. Pulling records based on time via REST APIs 2. Maintain ordering of incoming data // records are strictly ordered with topic-partition 3. Record deletion of data with tombstone records in the Kafka topic 4. Deduplication, if required 5. Realizing the schema of the records/tables and ingest them to Kafka topics with schema
- 37. General implementation tips 1. Connectors that communicate with external systems through REST clients a. Always retry in case of transient errors b. Make sure to log the responses c. …. 2. Rate limiting // Limitation from external systems 3. Task reconfiguration // To address changes in the external system 4. SMTs 5. Backward compatibility
- 38. Cloud readiness for Connectors 1. Rich Validations a. The Kafka Connect validation API needs to be fully and completely implemented so as to ensure proper integration with the Confluent Cloud UI. Any input provided by the user should be validated not only for correctness but also for completeness. For example, a validation should not just check that a host URL for an external system is well-formed, but also that the system is live and can be accessed with the user-supplied credentials. 2. ConfigProvider a. The connector should support injecting Credentials via the Apache Kafka ConfigProvider class. Confluent Cloud stores credentials in a secret store and injects it dynamically.
- 39. Confluent Hub 1. Build Connector 2. Check for license (restriction-free licensing, including Apache2.0) 3. Connector packaging 4. Submission
- 40. References and links a. Confluent Connector Dev Guide b. JDBC Connector Source Code c. JDBC Connector Documentation d. Kafka Connect Documentation e. Kafka Connect Architecture f. Confluent Hub
- 41. References and Links 5 Verified Integrations Program - Get Started confluent.io/verified-integrations-program/ • Verification Guides for Gold and Standard • Checklists • Templates
- 42. Thank you! Questions? Use the Q/A panel at the bottom
- 43. developer.confluent.io cnfl.io/online-talkscnfl.io/blog Resources are available using this icon Yashvardhan Khemani yashvardhan.khemani@hashedin.com Fahad Sheikh Fahad.sheikh@hashedin.com