Data lineage and observability with Marquez - subsurface 2020
0 likes839 views
This document discusses Marquez, an open source metadata management system. It provides an overview of Marquez and how it can be used to track metadata in data pipelines. Specifically: - Marquez collects and stores metadata about data sources, datasets, jobs, and runs to provide data lineage and observability. - It has a modular framework to support data governance, data lineage, and data discovery. Metadata can be collected via REST APIs or language SDKs. - Marquez integrates with Apache Airflow to collect task-level metadata, dependencies between DAGs, and link tasks to code versions. This enables understanding of operational dependencies and troubleshooting. - The Marquez community aims to build an open
![Marquez: Metadata collection
{
"type":"POSTGRESQL",
"name":"analyticsdbâ,
"connectionUrl":"jdbc:postgresql://localhost:5431/analyticsâ,
"description":âContains tables such as office room bookings.â
}
{
"type":"DB_TABLE",
"name":"room_bookingsâ,
"physicalName":"public.room_bookingsâ,
"sourceName":"analyticsdbâ,
"namespace":"datascience",
"fields": [...],
"description":âAll global room bookings for each office.â
}
02 Dataset
Source01](https://image.slidesharecdn.com/datalineageandobservabilitywithmarquezsubsurface2020-200807003708/85/Data-lineage-and-observability-with-Marquez-subsurface-2020-19-320.jpg)
![Marquez: Metadata collection
{
"type":"POSTGRESQL",
"name":"analyticsdbâ,
"connectionUrl":"jdbc:postgresql://localhost:5431/analyticsâ,
"description":âContains tables such as office room bookings.â
}
{
"type":"DB_TABLE",
"name":"room_bookingsâ,
"physicalName":"public.room_bookingsâ,
"sourceName":"analyticsdbâ,
"namespace":"datascienceâ,
"fields": [...],
"description":âAll global room bookings for each office.â
}
{
"type":"BATCH",
"name":"room_bookings_7_daysâ,
"inputs":[{"namespace":"datascience","name":"room_bookingsâ}],
"outputs":[],
"location":"https://github.com/jobs/blob/124f6089...â,
"namespace":"datascience",
"description":âWeekly email of room bookings occupancy patterns.â
}
03 Job
Source01
02 Dataset](https://image.slidesharecdn.com/datalineageandobservabilitywithmarquezsubsurface2020-200807003708/85/Data-lineage-and-observability-with-Marquez-subsurface-2020-20-320.jpg)
![Marquez: Metadata collection
{
"type":"POSTGRESQL",
"name":"analyticsdbâ,
"connectionUrl":"jdbc:postgresql://localhost:5431/analyticsâ,
"description":âContains tables such as office room bookings.â
}
{
"type":"DB_TABLE",
"name":"room_bookingsâ,
"physicalName":"public.room_bookingsâ,
"sourceName":"analyticsdbâ,
"namespace":"datascienceâ,
"fields": [...],
"description":âAll global room bookings for each office.â
}
{
"type":"BATCH",
"name":"room_bookings_7_daysâ,
"inputs":[{"namespace":"datascience","name":"room_bookingsâ}],
"outputs":[],
"location":"https://github.com/jobs/blob/124f6089...â,
"namespace":"datascienceâ,
"description":âWeekly email of room bookings occupancy patterns.â
}
03 Job
Source01
LINK SOURCE
LINK DATASET
02 Dataset](https://image.slidesharecdn.com/datalineageandobservabilitywithmarquezsubsurface2020-200807003708/85/Data-lineage-and-observability-with-Marquez-subsurface-2020-21-320.jpg)
![01 Job
v1
{
"type":"BATCH",
"name":"room_bookings_7_daysâ
"inputs":[{
"namespace":"datascience",
"name":"room_bookingsâ
}],
"outputs":[],
...
}
LINEAGE
JOBDATASET
Marquez: Metadata collection](https://image.slidesharecdn.com/datalineageandobservabilitywithmarquezsubsurface2020-200807003708/85/Data-lineage-and-observability-with-Marquez-subsurface-2020-22-320.jpg)
![{
"type":"BATCH",
"name":"room_bookings_7_daysâ
"inputs":[{
"namespace":"datascience",
"name":"room_bookingsâ
}],
"outputs":[],
...
}
JOBDATASET
Marquez: Metadata collection
02 Job
v2
{
"type":"BATCH",
"name":"room_bookings_7_daysâ
"inputs":[{
"namespace":"datascience",
"name":"room_bookingsâ
}],
"outputs":[{
"namespace":"datascience",
"name":"room_bookings_aggsâ
}],
...
}
LINEAGE
LINEAGE
01 Job
v1](https://image.slidesharecdn.com/datalineageandobservabilitywithmarquezsubsurface2020-200807003708/85/Data-lineage-and-observability-with-Marquez-subsurface-2020-23-320.jpg)
Data lineage and observability with Marquez - subsurface 2020
- 1. Data Lineage and observability Julien Le Dem CTO and co-founder Datakin @J_
- 2. AGENDA Intro to Marquez Marquez community 02 04 Why metadata?01 Airflow integration03
- 4. Need to create a healthy data ecosystem
- 5. Team interdependencies Team A Team B Team C
- 6. DATA â What is the data source? â What is the schema? â Who is the owner? â How often is it updated? â Where is it coming from? â Who is using the data? â What has changed? Today: Limited context
- 7. Maslowâs Data hierarchy of needs New Business Opportunities Business optimization Data Quality Data Freshness Data Availability
- 11. Metadata (Marquez) Ingest Storage Compute StreamingBatch/ETL â Data Platform built around Marquez â Integrations â Ingest â Storage â Compute Flink Airflow Kafka Iceberg / S3 BI
- 12. Marquez: Data model Job Dataset Job Version Run * 1 * 1 * 1 1* 1* Source 1 * â MYSQL â POSTGRESQL â REDSHIFT â SNOWFLAKE â KAFKA â S3 â ICEBERG â DELTALAKE â BATCH â STREAM â SERVICE Dataset Version
- 13. Marquez: Data model DbTable Filesystem Stream Job Dataset Job Version Run * 1 * 1 * 1 1* 1* Source 1 * â MYSQL â POSTGRESQL â REDSHIFT â SNOWFLAKE â KAFKA â S3 â ICEBERG â DELTALAKE â BATCH â STREAM â SERVICE Dataset Version
- 14. v1 v4Dataset v2 v4 v4 Job v1 Dataset v4 Job v2 Marquez: Data model â Debugging â What job version(s) produced and consumed dataset version X? â Backfilling â Full / incremental processing Design benefits
- 15. Marquez: Metadata collection How is metadata collected? â Push-based metadata collection â REST API â Language-specific SDKs â Java â Python Marquez Job Dataset+job metadata
- 16. â Centralized metadata management â Sources â Datasets â Jobs â Modular framework â Data governance â Data lineage â Data discovery + exploration Metadata Service Marquez: Design Marquez Core Lineage Search REST API ETL Batch Stream
- 17. Extensions datakin Lineage analysis Lineage collectionAPIs Integrations Client - side Metadata Core DB Graph Storage Marquez UI Listener Core API
- 18. Marquez: Metadata collection Source { "type":"POSTGRESQL", "name":"analyticsdbâ, "connectionUrl":"jdbc:postgresql://localhost:5431/analyticsâ, "description":âContains tables such as office room bookings.â } 01
- 19. Marquez: Metadata collection { "type":"POSTGRESQL", "name":"analyticsdbâ, "connectionUrl":"jdbc:postgresql://localhost:5431/analyticsâ, "description":âContains tables such as office room bookings.â } { "type":"DB_TABLE", "name":"room_bookingsâ, "physicalName":"public.room_bookingsâ, "sourceName":"analyticsdbâ, "namespace":"datascience", "fields": [...], "description":âAll global room bookings for each office.â } 02 Dataset Source01
- 20. Marquez: Metadata collection { "type":"POSTGRESQL", "name":"analyticsdbâ, "connectionUrl":"jdbc:postgresql://localhost:5431/analyticsâ, "description":âContains tables such as office room bookings.â } { "type":"DB_TABLE", "name":"room_bookingsâ, "physicalName":"public.room_bookingsâ, "sourceName":"analyticsdbâ, "namespace":"datascienceâ, "fields": [...], "description":âAll global room bookings for each office.â } { "type":"BATCH", "name":"room_bookings_7_daysâ, "inputs":[{"namespace":"datascience","name":"room_bookingsâ}], "outputs":[], "location":"https://github.com/jobs/blob/124f6089...â, "namespace":"datascience", "description":âWeekly email of room bookings occupancy patterns.â } 03 Job Source01 02 Dataset
- 21. Marquez: Metadata collection { "type":"POSTGRESQL", "name":"analyticsdbâ, "connectionUrl":"jdbc:postgresql://localhost:5431/analyticsâ, "description":âContains tables such as office room bookings.â } { "type":"DB_TABLE", "name":"room_bookingsâ, "physicalName":"public.room_bookingsâ, "sourceName":"analyticsdbâ, "namespace":"datascienceâ, "fields": [...], "description":âAll global room bookings for each office.â } { "type":"BATCH", "name":"room_bookings_7_daysâ, "inputs":[{"namespace":"datascience","name":"room_bookingsâ}], "outputs":[], "location":"https://github.com/jobs/blob/124f6089...â, "namespace":"datascienceâ, "description":âWeekly email of room bookings occupancy patterns.â } 03 Job Source01 LINK SOURCE LINK DATASET 02 Dataset
- 23. { "type":"BATCH", "name":"room_bookings_7_daysâ "inputs":[{ "namespace":"datascience", "name":"room_bookingsâ }], "outputs":[], ... } JOBDATASET Marquez: Metadata collection 02 Job v2 { "type":"BATCH", "name":"room_bookings_7_daysâ "inputs":[{ "namespace":"datascience", "name":"room_bookingsâ }], "outputs":[{ "namespace":"datascience", "name":"room_bookings_aggsâ }], ... } LINEAGE LINEAGE 01 Job v1
- 25. Airflow DAG DAG DAG DAG Marquez Lib. â Metadata â Task lifecycle â Task parameters â Task runs linked to versioned code â Task inputs / outputs â Lineage â Track inter-DAG dependencies â Built-in â SQL parser â Link to code builder (GitHub) â Metadata extractors Marquez: Airflow Airflow support for Marquez
- 26. DAG MarquezLib. Integration Marquez RESTAPI Capturing task-level metadata in a nutshell Marquez: Airflow Job Dataset Job Version Run Dataset Version * 1 * 1 1* 1* Source 1 * * 1 Airflow
- 27. â Open source: marquez-airflow â Enables global task-level metadata collection â Extends Airflowâs DAG class from marquez_airflow import DAG from airflow.operators.postgres_operator import PostgresOperator ... room_bookings_7_days_dag.py Marquez: Airflow Marquez Airflow Lib.
- 29. Marquez: Airflow t1=PostgresOperator( task_id=ânew_room_bookingâ, postgres_conn_id=âanalyticsdbâ, sql=âââ INSERT INTO room_bookings VALUES(%s, %s, %s) âââ parameters=... # room booking ) Operator Metadata Source01 new_room_booking_dag.py
- 30. Marquez: Airflow t1=PostgresOperator( task_id=ânew_room_bookingâ, postgres_conn_id=âanalyticsdbâ, sql=âââ INSERT INTO room_bookings VALUES(%s, %s, %s) âââ parameters=... # room booking ) Operator Metadata Source01 02 Dataset new_room_booking_dag.py
- 31. Marquez: Airflow t1=PostgresOperator( task_id=ânew_room_bookingâ, postgres_conn_id=âanalyticsdbâ, sql=âââ INSERT INTO room_bookings VALUES(%s, %s, %s) âââ parameters=... # room booking ) Operator Metadata 02 Dataset 03 Job new_room_booking_dag.py Source01
- 32. Marquez: Airflow new_room_bookings_dag.py top_room_bookings_dag.py Managing inter-DAG dependencies
- 33. Marquez: Airflow new_room_bookings_dag.py top_room_bookings_dag.py Managing inter-DAG dependencies b940314,1541624285,2 TSLOCATION ROOM b648485,1541501885,9 b648485,1541710685,4 public.room_bookings
- 34. Marquez API â Marquez standardizes metadata collection â Job runs â parameters â version â inputs / outputs â Datakin enables â Understanding operational dependencies â Impact analysis â Troubleshooting: What has changed since the last time it worked? Datakin leverages Marquez metadata datakin Lineage analysis Graph Integrations
- 35. Community04
- 37. Neutral â Not controlled by a company â Community driven Community â Build trust â Grow adoption â Everybody is on an equal footing Governance â Decision mechanisms â Becoming a maintainer â Code of Conduct Now part of the LF AI foundation
- 39. Thanks! <o/
- 40. Questions?