Data Discovery at Databricks with Amundsen
0 likes1,515 views
The document discusses data discovery challenges and the importance of metadata in decision-making processes for data professionals. It introduces Amundsen, an open-source data discovery and metadata platform designed to enhance productivity for analysts, scientists, and engineers by centralizing documentation and data lineage. The integration of Amundsen with Databricks aims to streamline data access, understanding, and usability across various data workloads.
Data Discovery at Databricks with Amundsen
- 1. Data Discovery At Databricks With Amundsen Tao Feng Tianru Zhou
- 2. Who Tao Feng ▪ Engineer at Databricks ▪ Co Creator of Amundsen ▪ Apache Airflow PMC ▪ Previously worked at Lyft, Linkedin, Oracle Tianru Zhou ▪ Engineer at Databricks ▪ Previously worked at AWS Elasticsearch
- 3. Data Discovery & Challenges
- 4. Data-Driven Decisions Analysts Data Scientists General Managers Engineers Experimenters Product Managers ● Axiom: Good decisions are based on data ● Who needs Data? Anyone who wants to make good decisions ○ HR wants to ensure salaries are competitive with market ○ Politician wants to optimize campaign strategy
- 5. Data-Driven Decisions 1. Data is Collected 2. Analyst Finds the Data 3. Analyst Understands the Data 4. Analyst Creates Report 5. Analyst Shares the Results 6. Someone Makes a Decision
- 6. Data Discovery Not Productive ● Data Scientists spend up to 30% of their time in Data Discovery ● Data Discovery in itself provides little to no intrinsic value. Impactful work happens in Analysis. ● The answer to these problems is Metadata / Data Catalog
- 7. Data Catalog to the rescue • Ease of documentation and discoverability ‒ Single searchable portal ‒ Display dependencies / lineages between data entities ( tables, dashboards) • Help to answer questions like: ‒ Where can I find data about ___? ‒ What is the context about the data? ‒ Who are the owners that I can ask for access? ‒ How is the data created? Is the data trustable? ‒ How should i use the data? Any sample query, statistics around the column? ‒ How frequently does the data refresh? ‒ ...
- 9. What is Amundsen • In a nutshell, Amundsen is an open-source data discovery and metadata platform for improving the productivity of data analysts, data scientists, and engineers when interacting with data. • Amundsen is currently hosted at Linux Foundation Data & AI (fromer LFAI) as its incubation project with open governance and RFC process. (e.g blog post)
- 12. Lineage between dashboards and dataset
- 13. Search for existing dashboards/reports
- 16. User Profile page
- 17. Announcement page • Plugin client to support new feature or new datasets
- 18. Central data quality issue portal • Central portal for users to report data issues. • Users could see all the past issues as well. • Users could request further context / descriptions from owners through the portal.
- 19. Data Preview • Supports data preview for datasets. • Plugin client with different BI Viz tools (e.g Apache Superset, Bigquery).
- 21. 5000+ Across the globe CUSTOMERS Lakehouse One simple platform to unify all of your data, analytics, and AI workloads The Data and AI Company ORIGINAL CREATORS
- 22. Databricks Lakehouse BI Reports & Dashboards Data Science Workspace Machine Learning Lifecycle Structured, Semi-Structured and Unstructured Data DELTA ENGINE Structured transaction layer High performance query engine
- 23. Internal dataset discovery at Databricks ● Static maintained wiki page for golden tables of the central workspace ● Metadata easily becomes stale ● Amundsen for the rescue!
- 25. Deployment(detailed) vpn Control plane amundsen ns Load balancer amundsen-frontend amundsen-search amundsen-metadata neo4j LB Data plane Databricks notebook Databricks job service Amazon RDS to store connections
- 26. Development Open source amundsen (git submodule) Private changes Private changes Base layer Layer m Layer n
- 27. Notebook version control Databricks private repo Databricks notebook Generate & grant access token Syncing changes
- 28. Metadata surfaced in amunden • Downstream/Upstream tables • Downstream jobs • Downstream users of the table • Job that writes the table • Writer of the table • Column stats • Dataset frequent users • Delta table extended metadata • Redash Dashboards • Sample data Lineage information Statistics Extended information
- 29. Lineage information Jobs that write the table Writer of table Main lineage info
- 30. What is table lineage
- 31. How is the lineage table generated? Raw lineage pipeline Raw -> processed lineage Usage_logs ReadEventTable (reads) WriteEventTable (writes) Insights_table Cleaning + workload aggregation Graph Read <-> Workload <-> Write Raw Lineage table With raw table paths dbfs:/user/hive/… → db.table String processing Paths → View conversion Get Delta metadata (Describe Extended) + String processing + heuristics Mount point → Blob path Get mount points (dbutils.fs.mounts()) + String processing Processed Lineage table With table Names
- 32. Statistics information Column statistics for numeric data type Frequent users Raw usage data also comes from usage_logs table analyze {table} compute statistics for column col1, col2 describe extended {db}.{table} `{column name}` Get column stats
- 33. Delta table extended metadata For delta table, we can run: describe detail table_name For delta table view, we can run: describe detail table_name Extract extended metadata
- 34. Notebook structure Open source delta_lake_metadata_extractor can be extended easily.
- 35. Notebook structure Step 1. Extract delta lake metadata + Publish to Neo4j Step 2. Publish to Elasticsearch
- 36. Notebook structure Step 3. Cleanup stale data
- 37. Redash dashboards All redash dashboards that use this table
- 38. Redash dashboards View in redash Copy button
- 39. Sample data Sample data tab Example:
- 40. WAU
- 42. Amundsen Open Source 1500+ Community members 2k+ Stars for the repo 30+ Companies using in production Also part of top 20 most popular OSS data projects in 2021 based on data council survey
- 43. Notable RFCs / PRs ● AWS Neptune metadata datastore (RFC#13) ● Mysql metadata datastore (RFC#019, RFC#021, RFC#023) ● Lineage frontend and backend (RFC#025, RFC#032) ● ETL push model paradigm (PR) ● Other rfcs could be found in here
- 44. Summary
- 45. Summary ● Solve data discovery challenges with Amundsen ● Integrate Amundsen with Databricks infrastructure ● Amundsen OSS adoptions significantly growing ●