Foundational Strategies for Trust in Big Data Part 2: Understanding Your Data
0 likes87 views
This document provides an overview of understanding data through data profiling. It discusses: 1. The five key steps to effective data profiling: defining how to analyze the data, what to review, what to look for, when to build rules, and what to communicate. 2. Common challenges with big data and new data types, and measurements for assessing data quality. 3. A case study of how British Airways leveraged data profiling and governance to ensure accurate customer data across multiple systems and improve analysis, marketing and service.
Foundational Strategies for Trust in Big Data Part 2: Understanding Your Data
- 1. Understanding Your Data Series: Foundational Strategies Trust in Big Data – Part 2
- 2. Webcast Audio • Today’s webcast audio is streamed through your computer speakers. • If you need technical assistance with the web interface or audio, please reach out to us using the Q&A box. Questions Welcome • Submit your questions at any time during the presentation using the Q&A box. • We will answer them during our Q&A session following the presentation. Recording and slides • This webcast is being recorded. You will receive an email following the webcast with a link to download both the recording and the slides. Housekeeping Arianna Valentini Product Marketing Manager
- 3. What You Will Learn Today • Quick refresh on ingredients of successful Big Data • Common challenges of Big Data and data profiling • The top 5 steps needed for effective data profiling • How another company saw success through data profiling • What you can do in the next 90 days to take action on DI Wrap up with: • Q&A 3
- 4. 4 Ingredients of Successful Big Data 1. Clear Business Case 2. Extract Data 3. Understand Data 4. Trace Lineage Data Governance
- 5. 80%of AI/ML projects are stalling due to poor data quality Dimensional Research, 2019 Big Data Needs Data Quality “Societal trust in business is arguably at an all-time low and, in a world increasingly driven by data and technology, reputations and brands are ever harder to protect.” EY “Trust in Data and Why it Matters”, 2017. The importance of data quality in the enterprise: • Decision making • Customer centricity • Compliance • Machine learning & AI 5 64%of IT executives have trouble finding and cleaning the right data for strategic data projects Sierra Venture, 2020 90%of executives are concerned about the how misused data can impact corporate reputation PWC, 22nd Annual Global CEO Survey, 2019
- 6. Understanding Your Data Data Profiling The set of analytical techniques that evaluate actual data content (vs. metadata) to provide a complete view of each data element in a data source. Provides summarized inferences, and details of value and pattern frequencies to quickly gain data insights. Business Rules The data quality or validation rules that help ensure that data is “fit for use” in its intended operational and decision- making contexts. Covers the accuracy, completeness, consistency, relevance, timeliness and validity of data. 6
- 7. Five Key Steps to effective Data Profiling These are not new, but good to reiterate in the context of Big Data: 1. How you want to analyze the data? 2. What should you review? (there's a lot of stuff) 3. What should you look for? (based on data “type”) 4. When should you build rules? (laser-focus; CDE’s) 5. What needs to be communicated? 7
- 8. 1. How do you want to analyze the data?
- 9. “ ” Never lead with a data set; lead with a question. Anthony Scriffignano, Chief Data Scientist, Dun & Bradstreet Forbes Insights, May 31, 2017, “The Data Differentiator”
- 10. Universal DQ best practices: Understand the End Goal • How does the business intend to use the data (i.e. what’s the use case)? • Empower users (“Who”) to gain new clarity into the core problem (“Why”) • What will the data be used for? • What defines the Fitness for your Purpose? Establish Scope • Ask the “right questions” about the use case and the data (not just “what” and “how”) • What data is relevant to the effort? • Big Data or other, you need to set boundaries for the work Understand Context • How does the business define the data? • What are the important characteristics and context of the data? • What are the Critical Data Elements? • What qualities will you need to address, or leave alone? • “High-quality data” definition will vary by business problem“If you don’t know what you want to get out of the data, how can you know what data you need – and what insight you’re looking for?” Wolf Ruzicka, Chairman of the Board at EastBanc Technologies, Blog post: June 1, 2017, “Grow A Data Tree Out Of The “Big Data” Swamp” 10
- 11. To Sample or not to Sample? Sampling helps with: • Data Integration • Source-to-target mapping • Data Modeling • Discovering Correlations When the focus is on the structure of the data • REMEMBER: your target is a statistically valid sample! • ~16k records gives you 99% confidence with a margin of error of 1% for 100B records • ~66k records gives you 99% confidence with a margin of error of .5% for same Full Volume needed with: • Data Quality • Data Governance • Regulatory Compliance • Finding Outliers and Issues with Content • “Needles in the haystack” When the focus is on the quality of or risks within the data • Focus on critical data elements and leverage tools that scale to data volume 11
- 12. Big Data at scale distributes data across many nodes – not necessarily with other relevant data! • Processing routines must apply same approach and logic each time • Implications for profiling, joining, sorting, and matching data, whether for enrichment, verification against trusted sources, or a consolidated single view Data Quality functions must be performed in a consistent manner, no matter where actual processing takes place, how the data is segmented, and what the data volume is. Scaling Data Quality best practices: Consistent processing at scale Source: HP Analyst Briefing 12
- 13. 2. What do you want to review?
- 14. Common Data Quality Measurements What measures can we take advantage of? 1. Completeness – Are the relevant fields populated? 2. Integrity – Does the data maintain an internal structural integrity or a relational integrity across sources 3. Uniqueness – Are keys or records unique? 4. Validity – Does the data have the correct values? • Code and reference values • Valid ranges • Valid value combinations 5. Consistency – Is the data at consistent levels of aggregation or does it have consistent valid values over time? 6. Timeliness – Did the data arrive in a time period that makes it useful or usable? 14
- 15. New data, new data quality challenges • 3rd Party and external data with unknown provenance or relevance • Bias in the data – whether in collection, extraction, or other processing • Data without standardized structure or formatting • Continuously streaming data • Disjointed data (e.g. gaps in receipt) • Consistency and verification of data sources • Changes and transformation applied to data (i.e. does it really represent the original input) New Data Quality Problems “34 percent of bankers in our survey report that their organization has been the target of adversarial AI at least once, and 78 percent believe automated systems create new risks, such as fake data, external data manipulation, and inherent bias.” Accenture Banking Technology Vision 2018 15
- 16. • Contextual visualizations • Value and pattern distributions • Attribute summaries and metadata • Sort and filter to quickly find data of interest • Detail drilldowns to any content Let Data Profiling guide you 16
- 17. 3. What should you look for?
- 18. Common Data Types What variances do you need awareness of? 1. Identifiers – data that uniquely identifies something 2. Indicators – data that flags a specific condition 3. Dates – data that identifies a point in time 4. Quantities – data that identifies an amount or value of something 5. Codes – data that segments other data 6. Text – data that describes or names something 18
- 19. 4. When do you build rules?
- 20. Focus on: • Critical Data Elements (data quality dimensions) • Policy-based conditions (e.g. regulatory compliance) • Correlated data conditions (e.g. If x, then y) • Filtering and segmenting data (refining evaluations; investigating root cause) Build Rules for Defined Conditions 20
- 21. • Validate critical requirements within or across data sources • Build common rules that can be readily tested and shared • Evaluate and remediate issues • Take action on incorrect data and defaults • Create flags for subsequent use in marking or remediating data • Filter result sets and export for additional use Benefits of Business Rules 21
- 22. 5. What should you communicate?
- 23. 23 Communicate! Culture of Data Literacy • “Democratization of Data” requires cultural support Program of Data Governance • Provide the processes and practices necessary for success Center of Excellence/Knowledge Base • Where do you go to find answers? • Who can help show you how?
- 24. • Annotate what you’ve found Annotate Results with Findings 24
- 25. British Airways Leveraging Data as a Critical Asset About • World’s leading international premium airlines • 33M passengers every year • 35,000+ employees • Fleet of 240 aircraft Goal • Ensure accurate data to support customer service, marketing, retention and loyalty • Implement enterprise-wide data governance Challenge • Data from multiple sources/systems, stored in many different formats • No enterprise standard for data quality • Point solutions led to varying levels of cleanliness, inefficiencies25
- 26. British Airways Results: Trusted data for improved analysis Solution • Trillium Data Quality Benefits Achieved • Trusted data for faster, better strategic and operational decision making • More effective marketing and better customer service 26
- 27. Looking at the Next 90 Days… • Make profiling actionable • You don’t know what you don’t know until you profile • Keep the 5 key questions top of mind! • Join us tomorrow for part 3 of our webinar series! 27
- 28. Questions?