Unleashing the value of metadata with Talend
4 likes5,176 views
The document discusses the importance of metadata management, highlighting the growing need for organizations to track and trace data due to regulatory requirements and the rise of big data and IoT. Talend offers a suite of tools designed to enhance metadata management, including data lineage, governance, and self-service access, catering to both business and technical users. With integrated solutions, Talend aims to support effective data management practices across various platforms and improve collaboration between IT and business lines.
Unleashing the value of metadata with Talend
- 1. Page 1 ©2016 Talend Unleashing the value of Metadata with Talend
- 2. Page 2 The Mandate for Metadata Management 2 Auditability Accessibility Agility IT needs to move faster and innovate more than ever More regulations and need for accountability need a map to track and trace the data journey Strong governance as a prerequisite to engage a wider audience of business users with self service
- 3. Page 3 e.g. in Banking: Risk and Finance data aggregation and reporting (Basel III BCBS 239) The need for governance and data lineage Impact of Risk Data & Technology Programs 21B of annual benefits Typical budget For Platforms 125M Investment >2% reduction Reduced capital and operations costs Source: Mc Kinsey & Company: Capturing value of BCBS 239 and beyond
- 4. Page 4 Why Now ? “The metadata management tool market will double in size in 2017, driven by the information of everything, proliferation of data from the Internet of Things (IoT), big data and data lake forces, and requirements for organizations to know what data they have, as information is recognized as an asset with an attached financial value.”
- 5. Page 5 The five pillars for managing Metadata with Talend Talend Studio Talend Metadata Bridge Talend Big Data With Cloudera Navigator Talend Data Preparation Talend Metadata Manager Metadata-driven, with zero coding Single point of control for Metadata Dependency checks and propagation Design once, propagate across any tools Apply change and enforce standards Accelerate migrations Data Lineage with depth on Hadoop Track, classify, and locate data & data flows Data Governance on Hadoop Data inventory for self-service access Auto-discovery with smart guidance Facet based search Risk and compliance with data transparency Increase agility across the data landscape Turn data into a business language (*) Apache Atlas integration planned in Talend 6.3
- 6. Page 6 Metadata-Driven Studio Transformation path from source to target Transformation lineage from target back to source Not just files & tables - Fined-grained metadata to the attribute - Automatically propagate changes Export & share https://help.talend.com/display/TalendDataFabricStudioUser Guide61EN/V.+Metadata+Management
- 7. Page 7 Talend Metadata Bridge • Available from Talend 5.6.2 • Over 100 connectors to import/export metadata from virtually any data-driven tools and platforms • Capabilities • Imports/Exports from design and modeling environments • Integrates with Databases, Big Data and development platforms • Reduces update & conversion costs and accelerates ETL offloading projects • Facilitates IT and Lines of Business collaboration for data flows specification
- 8. Page 8 Hadoop needs Metadata Management Lots of data Lots of Data Types Lots of Languages Lots of Frameworks One data, many replicas, many models
- 9. Page 9 + Hadoop meets Data Governance with Cloudera and Talend The first and only Cloudera Navigator integration that includes Spark Data Lineage
- 10. Page 10 A visual approach for data flow design, fully integrated with Cloudera Navigator Depth, breadth and reach for Hadoop Data Governance End-to-end data traceability, including upstream before the data reaches Hadoop Business users empowerment with metadata through business glossaries and data catalogs Deep control over your Hadoop flows Extend beyond Hadoop Reach new audience with Big Data
- 11. Page 11 Reclaim scattered files from users’ hard drives Single point of inventory and access for self service data Trace source, ownership, actions Improve accessibility, Auto- discover, recommend Data Inventory (from Talend Data Preparation 1.2) Govern Collaborate Control Guide
- 12. Page 12 Talend Metadata Manager • Available from Talend 6.1 (Dec, 2015) • Allows to harvest, govern, and share metadata across data platforms (BI, Big Data, modelling tools, ETL, etc.) • Provides : • Business Glossary • Shared definition and collaboration around data definition and policies • Enterprise Data Governance & Auditing • Data Flow Lineage and impact analysis, across systems and tools • Ensures Data Compliance • through model versioning, comparisons and reporting
- 13. Page 13 Talend Metadata Manager – Integrated Solutions IT and corporate servicesLines of Business Metadata Management Solutions Data Governance Solutions Any Enterprise Architecture from the Data Lake to the Data Warehouse Solutions to both business and technical users With any vendors/tools
- 14. Page 14 Talend Metadata Manager – The Big Picture For today’s evolving Enterprise Architectures integrating the Data Warehouse SemanticLineage Data FlowLineage Impact Analysis Data Warehousing Data Integration Business Intelligence Data Governance Data Standardization Enterprise Architecture Business Glossary Semantic Mapping Conceptual & Logical Models Design Patterns & Reusable Components Data Stores & MDM BusinessApplications BusinessReports and the Data Lake both
- 15. Page 15 Talend Metadata Manager – Metadata Harvesting http://www.metaintegration.com/Products/MIMB/SupportedTools.html
- 16. Page 16 Talend Metadata Manager – Enterprise Architecture
- 17. Page 17 Talend Metadata Manager – Metadata Browsing
- 18. Page 18 Talend Metadata Manager – Lineage & Impact Analysis
- 19. Page 19 Talend Metadata Manager – Data Model Diagrams
- 20. Page 20 Talend Metadata Manager – Data Governance ISO 11179 standard based Business Glossary Custom Attributes Glossary Bootstrap from the existing enterprise data model in ERwin, and more Bulk Change and Edit allowing for quick tabular editing and bulk changes to search results Semantic Mappings Governance Workflow provides a very flexible (customizable) workflow
- 21. Page 21 Talend Metadata Manager – Business Intelligence Tools BI Report Documenter Managed Business Intelligence: Live integration with BI Applications Talend MM application as a powerful multi-vendor BI Web Portal BI Report Documenter allowing users to get live definitions from any objects on reports, add them in place if missing, trace data lineage from any object in reports, find other related reports (even in other BI tools), produce live report specific glossaries (e.g. Tableau worksheet) Report Documenter Business Glossary Reuse Terms Create Terms
- 22. Page 22 Talend Metadata Manager – Data Modeling Tools Business Intelligence Tools Metadata Authoring Apps: Data Modeler / Documenter Data Modeler Business Glossary Data Stores Active Data Governance Data Standardization Automatic Live Updates of any data store schema change Reuse Terms Create Terms Forward Engineering to: - to self service BI like Tableau - or traditional BI Design Tools SAP BusinessObjects UNV/IDT IBM Cognos FM
- 23. Page 23 Metadata Harvesting Talend Metadata Manager – Talend DI StudioTalend MM Data Mapper Data Mapping Specifications Metadata Authoring Apps: Data Mapper Data Mapping Requirements Data Flow Implementation Active Data Governance Data Flow Lineage & Impact Analysis Metadata Comparision Requirement Compliance DI/ETL/ELT Tool Generation Data Mapping Design
- 24. Page 24 ©2016 Talend Unleashing the value of Metadata with Talend