Observability for Data Pipelines With OpenLineage
0 likes752 views
The document discusses the importance of metadata in data ecosystems and introduces OpenLineage, an open standard for lineage collection. It highlights the need for effective metadata management to avoid duplication and enhance collaboration across data pipelines, with an emphasis on integrations and observability in Spark. The document also outlines the core model and facets for metadata, detailing how OpenLineage and Marquez can facilitate understanding of data dependencies and support troubleshooting efforts.
Observability for Data Pipelines With OpenLineage
- 1. Data lineage and observability with OpenLineage Julien Le Dem, CTO and Co-Founder Datakin | Mai 2021
- 2. AGENDA ● The need for metadata ● OpenLineage - the open standard for lineage collection - and Marquez, its reference implementation ● Spark observability with OpenLineage
- 3. The need for Metadata 3
- 4. Building a healthy data ecosystem Team A Team B Team C
- 5. 5 Today: Limited context ● What is the data source? ● What is the schema? ● Who is the owner? ● How often is it updated? ● Where is it coming from? ● Who is using the data? ● What has changed? DATA
- 6. Maslow’s Data hierarchy of needs New Business Opportunities Business Optimization Data Quality Data Freshness Data Availability
- 8. OpenLineage contributors Creators and contributors from major open source projects involved
- 9. Purpose: Define an Open standard for metadata and lineage collection by instrumenting data pipelines as they are running.
- 10. Purpose: EXIF for data pipelines
- 11. Problem Before: ● Duplication of effort: Each project has to instrument all jobs ● Integrations are external and can break with new versions ● Effort of integration is shared ● Integration can be pushed in each project: no need to play catch up With Open Lineage
- 12. Open Lineage scope Not in scope Backend Integrations Metadata and lineage collection standard Warehouse Schedulers ... Kafka topic Graph db HTTP client Consumers Kafka client GraphDB client ...
- 13. Core Model: - JSONSchema spec - Consistent naming: Jobs: scheduler.job.task Datasets: instance.schema.table 13
- 14. 14 Protocol: - Asynchronous events: Unique run id for identifying a run and correlate events - Configurable backend: - Kafka - Http Examples: ● Run Start event ○ source code version ○ run parameters ● Run Complete event ○ input dataset ○ output dataset version and schema
- 15. 15 Facets ● Extensible: Facets are atomic pieces of metadata identified by a unique name that can be attached to the core entities. ● Decentralized: Prefixes in facet names allow the definition of Custom facets that can be promoted to the spec at a later point.
- 16. Facet examples Dataset: - Stats - Schema - Version - Column level lineage Job: - Source code - Dependencies - params - Source control - Query plan - Query profile Run: - Schedule time - Batch id
- 17. 17
- 18. Metadata: Ingest Storage Compute Streaming Batch/ML ● Data Platform built around Marquez ● Integrations ○ Ingest ○ Storage ○ Compute Flink Airflow Kafka Iceberg / S3 BI OpenLineage
- 19. Marquez: Data model Job Dataset Job Version Run * 1 * 1 * 1 1 * 1 * Source 1 * ● MYSQL ● POSTGRESQL ● REDSHIFT ● SNOWFLAKE ● KAFKA ● S3 ● ICEBERG ● DELTALAKE ● BATCH ● STREAM ● SERVICE Dataset Version
- 20. API ● Open Lineage and Marquez standardize metadata collection ○ Job runs ○ Parameters ○ Version ○ Inputs / outputs ● Datakin enables ○ Understanding operational dependencies ○ Impact analysis ○ Troubleshooting: What has changed since the last time it worked? Datakin leverages Marquez metadata Lineage analysis Graph Integrations
- 22. 22 Spark java agent spark.driver.extraJavaOptions: -javaagent:marquez-spark-0.13.1.jar={argument}
- 23. Metadata collected 23 Lineage: inputs/outputs Data volume: row count/byte size Logical plan
- 24. Lineage model 24
- 25. Lineage Example across jobs
- 26. 26 Example of OpenLineage metadata usage: Data volume evolution
- 27. Join the conversation OpenLineage: Github: github.com/OpenLineage Slack: OpenLineage.slack.com Twitter: @OpenLineage Email: groups.google.com/g/openlineage Marquez: Github: github.com/MarquezProject/marquez Slack: MarquezProject.slack.com Twitter: @MarquezProject