OpenMetadata Spotlight - OpenMetadata @ Thndr by Fizza Abid
0 likes157 views
The document outlines Thndr's use of OpenMetadata for data cataloging and governance, highlighting the challenges faced in metadata visibility, data quality, and lineage. Thndr chose OpenMetadata due to its open-source nature, feature set, and community support, implementing it on AWS with a small team. Key features include automated data quality alerts, a centralized metadata repository, and role-based access control for enhanced data governance.
OpenMetadata Spotlight - OpenMetadata @ Thndr by Fizza Abid
- 1. OpenMetadata Community Forum August Data Cataloging and Governance Using OpenMetadata at Thndr
- 2. Data Engineer (Data Platform Engineer) ● 5+ years of experience in Data Engineering ● Worked on ETL, Data Quality, Data Governance, platform & infrastructure deployments. ● Analyze optimizations in the data platform and analytics Fizza Abid
- 3. Agenda ● What does Thndr do? ● What is the OpenMetadata Architecture at thndr? ● What were the Metadata Challenges at Thndr? ● Why did we choose OpenMetadata? ● What are use cases for using OpenMetadata? ● How did we implement OpenMetadata at Thndr?
- 4. What does Thndr do? Push on Education No Barriers Investment Supermarket Relevant & Intuitive Users can open and manage accounts without visiting branches or access to restrictive capital and can fund their accounts easily Providing access to all relevant investment products whether local or abroad Equipping out users with everything they need for a successful investing journey Relevant interface and hand-holding focused experience
- 5. Why invest on Thndr?
- 6. What has Thndr Achieved?
- 7. ● Buy and Sell Stocks ● Invest in Mutual Funds ● Invest in Gold ● Instant Top Ups and withdrawals Features of Thndr
- 8. ● Automated Data Quality Alerts ● Visibility about metadata (freshness, volume, schema changes, etc.) ● Manual detection of PII columns ● Business or Technical Glossary ● Centralized metadata repository Metadata Challenges at Thndr Lineage Data Quality Metadata Visibility Manual Pll Detection
- 9. Why did we choose OpenMetadata? Open Source OpenMetadata is open-source and it can be deployed on EKS or docker easily 1 Features The features that they’re offering are better than paid solutions and in a single place, we can find metadata management, lineage, data quality, etc. 2 Community Support OpenMetadata community is support on reaching out on slack, they answer your questions instantly 3
- 10. OpenMetadata Architecture at Thndr
- 11. ● Deployed OpenMetadata on EC2 machines using Docker image on AWS ● We used EC2 Machines on AWS ● Rotate JWT tokens when deploying OpenMetadata ● SSO deployment for an added layer of security Deployment
- 12. ● We have small Team of 2 Data Engineers, one data platform engineer, and Team of 3 Data Analyst ● Business Users can know what particular column in data products means using OpenMetadata Glossary Feature ● Data Quality checks are implemented by Data Engineers and Data Analyst ● We have small number of data quality test yet but we’ll be putting it on more tables. ● Most of the checks are done for Batch ETL like checking duplicates, freshness of data, etc. OpenMetadata Usage at Thndr
- 13. OpenMetadata Usage at Thndr
- 14. ● Search in the explorer based on table name, column name, schema name ● inbuilt data cataloging feature with many different datasets and tables, each with varying access levels for different users ● Data catalog provides a single, unified user experience for quickly discovering these datasets. ● Metadata, DDLs, views, and stored procedures will be available and can be explored through data catalog. Data Discovery
- 15. Data Cataloging
- 16. Data Classification ● OpenMetadata uses machine learning to auto classify PII columns into sensitive and non-sensitive. ● Also provide custom tagging into personal or special category
- 18. ● Data Lineage in OpenMetadata is available on either observability layer or column layer ● Data lineage shows the complete journey and transformation of data from its origin to its final destination. ● Export data lineage in excel Data Lineage
- 19. Data Quality ● Data Quality Feature in OpenMetadata allows to run custom test cases based on the table or columns. ● Custom SQL Query for data quality testing like if you want to compare values in multiple tables or have any dynamic logic. ● Data quality test include match column name to certain regex, value in between, not null, etc.
- 20. Data Quality
- 21. Data Quality
- 22. Data Quality
- 23. ● Role based access control team wise, department wise, and roles include Data Consumer, Data Steward, etc. ● Attribute based access control ● Granular level of access control on resources Data Governance