Ensuring data quality with lakeFS
- 1. Ensuring quality in a data lake environment with lakeFS August 18, 2021 By Paul Singman
- 2. o Fast Development & Effective Collaboration o Intelligent Data Deployment o Straightforward Error Resolution Important Factors for Quality
- 3. The Data Lake Advantage
- 4. 1. Scalable and cost effective The Data Lake Advantage
- 5. 1. Scalable and cost effective 2. Highly accessible The Data Lake Advantage
- 6. 1. Scalable and cost effective 2. Highly accessible 3. High throughput The Data Lake Advantage
- 7. 1. Scalable and cost effective 2. Highly accessible 3. High throughput 4. Rich application ecosystem The Data Lake Advantage
- 8. Data Lake Architecture Reference Events Data Operations Data OBJECT STORAGE Data Processors Analytics Engines Data Viz Data Exploration
- 9. o Inability to experiment, compare and reproduce Example: test version or algorithm upgrade o Difficult to enforce best practices Example: schema, format enforcement o Hard to recover from production errors Example: delete files in the object store The Data Lake Challenge
- 10. Atomic versioned data lake on top of object storage In a perfect world
- 12. Important Factors for Quality o Fast Development & Effective Collaboration o Intelligent Data Deployment o Straightforward Error Correction
- 13. o Fast Development & Effective Collaboration o Intelligent Data Deployment o Straightforward Error Correction Important Factors for Quality from lakefs_client.client import LakeFSClient client = LakeFSClient(config) client.branches.create_branch(repository=‘my-repo’, branch_creation= models.BranchCreation(name=‘experiment2’, source=‘experiment1’) ) df1 = sc.read.parquet("s3a://my-repo/experiment-1/events/by-date") df2 = sc.read.parquet("s3a://my-repo/experiment-2/events/by-date") df1.group_by("...").count() df2.group_by("...").count() // now we can compare the properties of the data itself
- 14. o Fast Development & Effective Collaboration o Intelligent Data Deployment o Straightforward Error Correction Important Factors for Quality from lakefs_client.client import LakeFSClient client = LakeFSClient(config) client.refs.merge_into_branch( repository=‘my-repo’, source_ref=‘experiment1’, destination_branch=‘main’ ) merge changeset: ✓ 001.parquet ✓ 002.parquet random.csv new- data-1 main
- 15. o Fast Development & Effective Collaboration o Intelligent Data Deployment o Straightforward Error Correction Important Factors for Quality lakectl branch commit lakefs://example-repo@stream-1-branch lakectl branch revert lakefs://example-repo@stream-1-branch --commit dd8a60d5ef70809
- 16. Data Consumption Data Sources !"#$%&'(&)*$ Manageability & Resilience Layer
- 17. Integrates with your existing tools Object Store Streaming Data Batch Jobs Data Visualization MLOps Query Engines Data Quality
- 19. Architecture lakeFS Metadata Manager lakeFS S3 Gateway Spark/Presto/EMR, ... lakeFS Hadoop Client Other Data Consumers/Producers Object store (S3) rocksDB metadata Deduplicated Data Objects
- 20. Data Format Commit Meta range ID: ... Master Commit ID: ... 8-12MB .sst files, containing continuous sorted keys .sst file, pointing to range files Further Reading
- 21. Additional Resources Getting started Check out the docs Join the lakeFS Slack Channel Contribute and star the repo
- 22. Thanks! Join the community einat.orr@treeverse.io GitHub