OpenMetadata Webinar on Custom Connectors
1 like363 views
The document outlines the process and considerations for integrating internal systems with OpenMetadata using custom connectors, focusing on ingestion frameworks and metadata management. It highlights the pros and cons of using schema-first approaches, APIs, and SDKs in various programming languages. Additionally, it provides links to resources and demos for creating custom connectors and managing metadata workflows.
OpenMetadata Webinar on Custom Connectors
- 2. ● Schema-First ● Metadata APIs ● JSON Schema
- 4. Company Context ● Internal systems built in-house ● External systems ● No out of the box solution
- 5. + Flexibility - Abstraction API Calls Python & Java SDKs Ingestion Framework How do we bring Internal Systems into OpenMetadata? - Flexibility + Abstraction
- 6. Creating a Table with the API
- 7. Creating a Table with the API More info at https://docs.open-metadata.org/swagger.html PROS ✅ CONS ❌ ● Close to no requirements ● Verbose ● No safeguards
- 8. Creating a Table with the Python SDK
- 9. Creating a Table with the Python SDK Java - https://www.jsonschema2pojo.org/ Python - https://github.com/koxudaxi/datamodel-code-generator Typescript - https://www.npmjs.com/package/quicktype
- 10. Creating a Table with the Python SDK More info at https://docs.open-metadata.org/sdk/python PROS ✅ CONS ❌ ● Metadata standard ● Helper methods ● Language-specific
- 11. Creating a Custom Connector Source Processor Sink ● Ingestion ● Lineage ● DBT ● Profiler ● …
- 12. Creating a Custom Connector What are the Benefits? ● Only focus on extracting metadata ● The Ingestion Framework manages how to schedule and write the metadata ● Run and schedule the ingestion directly from the UI
- 13. Creating a Custom Connector PROS ✅ CONS ❌ ● Focus on business logic ● Integrated with the full Ingestion Framework ● Less flexibility
- 14. Read a CSV Validate its contents Transform to the OM standard Send it to the next step in the workflow Demo: Creating a Custom Connector
- 15. DEMO: Creating a Custom Connector https://github.com/open-metadata/openmetadata-demo/ tree/main/custom-connector