Working With Different Kinds of Data
0 likes804 views
The document discusses the challenges of integrating data from multiple sources. It notes that continuously streamed data sources can influence business analytics by impacting customer satisfaction, identifying opportunities, and more. However, integrating these different data sources poses challenges like ensuring entity identifiability across datasets, dealing with limited or no existing data governance, addressing potential editorial biases, and handling missing metadata. The document provides examples and strategies for profiling data elements, determining if they are structurally and semantically conformable, and leveraging metadata to enable successful data integration.
Working With Different Kinds of Data
- 1. Integrating Data from Multiple Sources 2015-02-26 David Loshin Knowledge Integrity, Inc. loshin@knowledge-integrity.com © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 1
- 2. Ingesting Data from Multiple Sources • Continuously streamed data sources may influence business performance analytics: – Influence customer satisfaction – Expose opportunities for revenue generation – Identify brand risk – Flag fraud and abuse – Improve customer profiling and customer experience © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 2
- 3. Challenges • Entity identifiability • Limited or no data governance • Editorial bias • Absence of metadata © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 3
- 4. Entity Identifiability • Recognizing and resolving identities is challenging for static, complete data sets • Entity identifiability becomes more challenging when merging static and streamed information: – Entity attribute identification – Entity recognition – Identity resolution – Linkage across data sets © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 4 Is this the same guy?
- 5. Limited or No Data Governance • Little or no knowledge of – Defined data quality criteria – Edits or controls – Chain of accountability • Limited shared definitions – Typically tabular data dictionaries with nondescript definitions • Harvested data has no discernable lineage – Completely devoid of context or production chain © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 5
- 6. Editorial Bias • Creating data sets for external consumption involves editorial decisions and biases • Choices are made about – The physical structure of the data values – Which data elements are included – Which are excluded from the final artifact © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 6 Selection criteria
- 7. Absence of Metadata • Numerous data sources have little or no metadata at all – Dynamically harvested tabular data – Scraped data – Human-generated content – Automata-generated content – Unstructured data artifacts – Other data artifacts (graphics, images, video, audio, etc.) © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 7
- 8. Example: Healthcare Provider Data • NPPES Provider First Line Business Mailing Address • Definition: – “provider’s first line business mailing address” • Open Payments Recipient_Primary_Business _Street_Address_Line_1 • Definition: – “The first line of the primary practice/business street address of the physician or teaching hospital (covered recipient) receiving the payment or other transfer of value.” © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 8 • Is “provider” the same as “recipient”? • Are these conformant data elements? • Actually it turns out that the Open Payments data element is sourced from the NPPES data set!
- 9. Preparing to Integrate • Infer the source data sets metadata • Determine if the data element inventories are structurally conformable • Determine if the data element inventories are semantically conformable © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 9
- 10. Inferring Metadata Using Profiling • Analysis of data sets, records, data elements, and data values to – Infer data element types and sizes – Identify reference value domains – Make educated guesses about intent/meaning © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 10 Attribute First d 4 6 y Last f 6 2 h Street d 4 7 n City a 0 2 o State Value Count A 12000 I 10000 L 7655 X 3208 N 120 M 8 Profiling
- 11. Conformable Data Elements • Data elements are conformable if – Share the same data element concept – Share the same value domain – Share the same definition and semantics © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 11 • These two data elements are conformable if their definitions are the same! CountryOfOrigin 2-character IDO 3166 Country Code CountryOfManufacture 2-character IDO 3166 Country Code
- 12. Using Metadata to Test Conformability • Inferred structural metadata provides the first cut at determining whether two data elements are conformable • Introduce internal governance and management around external metadata – Use a metadata repository to capture inferred metadata – Define policies for identification, assessment, documentation, and use of external data sources – Institute stewardship for each external data source for process management, validation, and maintenance • Select a metadata tool that provides – Enterprise-wide metadata visibility – Integration with data assessment tools – Historical lineage for metadata capture – Collaboration among data consumers © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 12
- 13. Questions & Suggestions • www.knowledge-integrity.com • www.dataqualitybook.com • www.decisionworx.com • If you have questions, comments, or suggestions, please contact me David Loshin 301-754-6350 loshin@knowledge-integrity.com © 2015 Knowledge Integrity, Inc loshin@knowledge-integrity.com (301) 754-6350 13
- 14. EMBARCADERO TECHNOLOGIESEMBARCADERO TECHNOLOGIES Joy Ruff Product Marketing Manager | ER/Studio Joyce.Ruff@embarcadero.com ER/Studio Team Server Overview
- 15. EMBARCADERO TECHNOLOGIES Keeping pace with the rapid growth of data, change and compliance Evolving Database Ecosystems Volume, Velocity, Variety Agile Development Cycles Maximizing IT Infrastructure ComplianceLimited Resources Database Professionals Need the Right Tools 15
- 16. EMBARCADERO TECHNOLOGIES Share Models & Metadata with Business & IT 3 Team Server ER Repository Modeling Teams • Business Analysts • Executives • App and DB Developers • Data Stewards • DBAs
- 17. EMBARCADERO TECHNOLOGIES • Powerful enterprise glossary & metadata collaboration • Integrate key business terms and definitions with business systems • View, store, and manage a single source of business definitions • Attach business policies to daily workflows with contextual alerts and tips
- 18. EMBARCADERO TECHNOLOGIES The Power of Unlimited Involvement • Use business terms to easily locate and relate information assets • Maintain enterprise glossaries, terms, and underlying metadata in a central interface • Enable a consistent flow of information and collaboration around data management 18 Contributors Business Architecture IT Definition Structure Deployment Syndication Collaboration Consumers Executive Analyst Developer Integration
- 19. EMBARCADERO TECHNOLOGIES Benefit of Relating Metadata to Models • Expand the depth of information by accessing the underlying framework 19 • Models and terms seamlessly integrate to one another
- 20. EMBARCADERO TECHNOLOGIES The Primary Resource for Data Information 20 • Manage a single source of business definitions in an enterprise glossary • Avoid the issue of information stagnation • Improve productivity and accuracy in data analysis, application, BI and ETL development
- 21. EMBARCADERO TECHNOLOGIES Data Source Registry
- 22. EMBARCADERO TECHNOLOGIES Unified Glossary and Terms 22
- 23. EMBARCADERO TECHNOLOGIES Empowering the Organization 23 ! ed!to!developing!industry7leading!tools! !for!over!20!years.!!Our!ER/Studio*Team* rney,!offering!modeling!and!metadata! s!users!gain!visibility!to!existing!data! e!critical!decision7making!assets!they!can! ortal!to!be!useful,!you’re!going!to!love! n(( Team( Server( Core( Portal( itions!with!data! b!assets!into!daily! ! ! Limit the level of confusion by centralizing glossaries, terms, and object relationships • Discuss and add to the development of models and metadata • Track and gain insight into who and what information has changed in the environment
- 25. EMBARCADERO TECHNOLOGIES The Right Tools are Everything Discover the Benefits of the Ultimate Cross-Platform Database Tools 25
- 26. EMBARCADERO TECHNOLOGIES Thank you! • Learn more about the ER/Studio product family: http://www.embarcadero.com/data-modeling • Team Server Hosted Trial: http://www.embarcadero.com/products/er-studio/team- server-hosted-trial • To arrange a demo, please contact Embarcadero Sales: sales@embarcadero.com, (888) 233-2224 26