Establishing a Strategy for Data Quality

Editor's Notes

• #2: I am a Principal Consultant with Database Answers Ltd For the past 3 years I have been the Data Architect with the London Borough of Ealing
• #3: Why is Data Quality important ? Gartner says “Fortune 1000 enterprises lose more money due to data quality issues than they spend on DW and CRM” Forrester “In recent discussions, not one out of 30 companies expressed confidence in their Customer Data” TDWI says “DQ Problems cost American businesses more than $600 billion dollars a year” Local Authorities A recent Report from the Audit Commission on DQ in Liverpool City Council emphasises the importance of DQ in Performance Indicators Liverpool has a Performance Management Database (PMD) and The Audit Commission recommends training and in DQ “ identification of staff with DQ Responsibility which should be specified in Job Descriptions”. URL - http://www.liverpool.gov.uk/Images/tcm21-116883.pdf Identifying the Infrastructure (10 Slides) – Start 9:40 am- Data Architecture Based on my fifteen years experience Focus in particular on DQ Data Architectures Data Metrics Setting Quality Control Initiative (7 Slides) – Start 9:50 am - Tools Data Arch helps us to choose Tools - Let’s look at choosing Tools Developing plans to enrich Quality (10 Slides) – Start 10:00 am – Data Platform (Engage) Engage with the Business - Data Platform Getting started (7 Slides) – Start 10:10 am – Combine Organisation and Technology Look now at Organisation aspects and how technology and business must be combined Business Drivers - Roles and Responsibilities - Data Quality Audit
• #4: The Data Warehousing Institute says :- “ Data quality is a complex concept that encompasses many data management techniques and business-quality practices, applied repeatedly over time as the state of quality evolves, to achieve levels of quality that vary per data type and seldom aspire to perfection .” Wikipedia says :- “ Data is high quality “if they are fit for their uses” “ Achieve degree of excellence” (GIS Glossary) “ Covers the state of completeness, validity, consistency, timeliness and accuracy that makes data appropriate for a specific use ” (BC Govt) High quality if “ Good enough ” – which at first sounds bad but then you realize it’s acceptable. Barry’s Definition – “Fit for Purpose” Eg Council Tax are not so concerned with gender and Date of Birth , whereas Social Service are very concerned, and need to have 100% confidence in the data. This leads to the idea of specific Systems being the authority for specific Data Items, and the Custodian of those Systems being the Owner or Data Steward. Called “ System of Record ” The remainder of my Presentation will discuss the implications of Data being “Fit for Purpose” and if it’s not, how do we achieve it. As we will see, this will involve both technical aspects and organizational aspects. BUT, as we move towards establishing Data Marts , we need join values and get consistent PIs, Therefore we need all data to be 100%. In other words, if the purpose is Data Marts or Data Warehouses, then DQ has to be 100% across the board.
• #5: Framework Two aspects – data-related and organisation-related Both must be in sync. As-Is and To-Be Migrating from now to future preferred state Roles Everybody has to agree and understand their roles Let’s look at some typical DQ Projects …
• #6: Barclays (1993) From 1 System to 1 System in batch Simple migration from a commercial system to a new replacement system Involved cleaning-up Customer data, products and orders, (invalid dates) – my first realisation that people will always enter bad data . Required Users/Owners, Signoff/Cleanup,transformation – I was introduced to vulgar addresses . Barclays (1998) From 6 Systems to 1 System in real-time Migration from Product-oriented Systems to Customer-facing O-O approach as part of a move to ‘Single View of ‘ Customer-friendly strategy Centrica From 30 Systems to 1 System in batch – overnight loading Corporate Data Admin 30 Systems holding Customer Data Systems Data Quality Audit for AA CDI and CMI with clean-up en route Cisco From 15 Systems to 1 System in batch – one at a time Limited Scope – migrating Customer and Products data from 15 Eastern European countries to a single Corporate Data Warehouse. Eg Polish , to UK Comma in money to full stop, and Polish Job Titles to Corp std Validate dates – starting, ending, etc. Realise some DQ Rules are common sense (start end end dates) and some are business-specific (eg Job Titles Ref Data) Ealing From 6 Systems to 1 System in batch for Debtors Data Mart and batch and real-time for SMPL Data Architect – Corporate concerns, eg CMI, DPA, Debtors Data Mart requiring consolidation across 8 Systems. Street Environment – focussing on Service-specific data issues - from 4 Systems to 1 System in real-time (with PDAs) including batch validation of Schedules and real-time cleanup, eg PDA readings outside Ealing – look wrong but are acceptable – need user sign-off.
• #7: This was my first formal experience in Data Quality. I was very lucky because it was a great opportunity where I was presented with a Problem and asked to come up with a Solution. The Solution I produced involved many features that are always important in DQ Projects. These include Data Owners, User Involvement and sign-off, Incremental advances, a Library of Validation Rules in English and SQL , and so on. This Report shows :- Date and Time of Run Error count Test Description For example, ‘ orphans ’ – Client Debts without a Client.
• #8: 1) Data Governance Top-down, SOPs Enterprise Standards 2) DQ Architecture From basic to advanced and real-time DQ Metrics and Criteria Users/roles/results/ Fields/metrics/Required Stats 3) Profiling Data Profiling is a good start because you get familiar with the data and the kinds of errors. 4) Data Owners Must get buy-in Provide time and commitment Define/agree rule 5) Choose Tools Can waste a lot of time in manual work, and by choosing the wrong tool. My experience is to start small and migrate to a more powerful Tool with a clear objective idea of where you want to get to – this also helps the business case Now let’s look at some Data Architectures for DQ work …
• #9: Entry-level Based on my first exp 15 years ago at Barclays Bank DIY – no tools – clear scope and limited budget Rules in SQL Start small, evolve, become familiar with data Same approach as later at Cisco – clearly defined scope (and deliverables) with no real extension. And at Ealing (change Hats) (and at Haringey beforehand) – as a starting-point and Proof-of-Concept
• #10: Centrica British Gas and the AA 30 Systems holding Customer data Required Enterprise Data Model Create Library of Scripts to check Rules The Library held Standards – eg Customer Categories, Default Dates, Rules for checking against Corporate LOV’s Build up Reports with User Sign-Off Reports included Audit findings, sign-off by Users Leading to agreed level of DQ
• #11: At Ealing (and at the London Borough of Haringey) Multiple Applications with Single View of Customers Data Hub is linked to a Customer Master Index The Data Quality Engine is a more powerful Tool, such as Data Flux A Data Dictionary essential – published over Intranet for sign-off and reference Rules for Validation and Transformation
• #12: We can see some elements of the future in the present and extrapolate The DQ Professional wants to say “My Data is over there, Analyse it and give me the Results” This a big jump from 1993 and reflects Web Services and even Web 2.0 approach. At Ealing we are using Web Services in our SMPL Performance Reporting Mobile Application at Ealing which transmits data to a Consolidated Database. Data Quality in real-time … is our next topic …
• #13: Validate in Batch Every 3 months we load Schedules as batch data into the Database We validate for reasonableness (eg volumes within predicted ranges) and specifics like dates Also Validate Data on Entry The data is checked in Real-Time when it’s added to the Database – eg that location is in Ealing because Inspectors sometimes park outside. PDA with GPS locations outside Ealing Perception can be as important as reality, which needs to be taken into account
• #14: This dashboard is from a company called Acuate It is a vision of the Future. I use my Web Site to demo State-of-the-Art and this is one of the facilities. The 3 circles show the effect of automatic matching for a DQ Test Run 2 = 3 (post-automatic=final) This brings us to the topic of Metrics …
• #15: A Dashboard needs metrics eg Customers – “How many Customers do we have ?” Clear Definitions – eg “What exactly is a Customer ?” At Centrica, a Customer could be anyone who showed an interest – eg call for details of special offers “ Need to count Customer Ids and match duplicates However, a Report in the Government Computing Magazine in January , 2004, states :- No Government anywhere in the world has successfully introduced and maintained an authenticated Unique Identifier for each Citizen. Many claim success but cannot provide hard evidence. In the UK, there are 81million NI numbers but only 60 million eligible citizens ”. We have 17 John Bevans Easy to Measure Matching name and addresses Need ability to store aliases Relevant to the Business Eg in Ealing, people drop-in, phone, email, respond to “Contact Us” on the Web Site But how important is it to the business to match them ? Fraud detection can make it important SMPL Performance Reports are based on KPIs which need clean data, which we will see later. Now let’s turn to Setting a Quality Control Initiative, which includes Choosing Tools …
• #16: Objectives of the Initiative Establish the mindset Get started and establish sound foundation Define some specific results - % overall data quality or Customer Matching and % duplicates Quick Wins - Current problems – where does the shoe hurt ? eg Chief Exec asked – “What is Ethnic Breakdown in Ealing ?”, “How many people work for Ealing Council ?” This highlighted the need for standards , such as Classifications of Ethnic Origins, Work basis – Full-Time, Part-Time, Contract, Agency Staff, Temporary, and so on. DQ Architecture Help define Requirements Requirements help evaluate Tools and Vendors Let’s look at some Tools …
• #17: I have done this myself at some major organisations, including the AA, Barclays Bank, Cisco and Ealing. For example, at Cisco , I was migrating Customer data from 15 European countries to one Corporate Oracle Database Therefore the use of Templates was clearly the correct approach. In fact, one Template for a generic Customer , taking data from Access Databases, commercial Packages , Excel spreadsheets and so on. I used Oracle’s PL/SQL to translate commas in Polish money to full stops in UK money.
• #18: Ab-Initio - in wide use for profiling, which is a very important function. - eg at the AA – ranges of Membership Start and End Dates. InfoShare Clearcore (London) Very useful at Ealing for Customer Matching “ Single Citizen View ” - http://www.infoshare.ltd.uk/solutions/single_citizen_view.html Case Studies are available for Local Authorities InSource (Reading) Gets ticks in lots of boxes Business Rules De-Duping Repository Single Version of the Truth *** Web Form Innovator **** Plus Based in Reading
• #19: At first, I thought ‘Great, let’s choose the Leaders’. Then I realised Gartner speaks to the needs of its subscribers , who are Blue Chips with appropriate budgets such as £250K for DQ Software. Niche players have a lot to offer under the right circumstances. There might be more of those than Blue-Chips Vendors ! Getting started is better with Niche players. The Leaders are not Niche players and Niche players are not Leaders However, Niche players do have a part to play – eg InfoShare – cost and functionality tailored exactly to the need. – eg Customers and De-duping. Early Adopters not catered for by Magic Quadrant Gartner Leader’s Quadrant DataFlux (a Leader) - we had discussions at Ealing for Proof-of-Concept work, but couldn’t establish a starting-point Business Objects and Cognos volunteered. Data Foundations (a ‘Cool Leader’) Gets ticks in lots of boxes - Universal Data Hub, MDM Methodology, Ref Data Mgt. Flexible Data Model and a Registry ( ISO 11179 compatible with MetaData Registry) Trillium (a Leader) – I used Trillium at Centrica for Customer Name and Address matching
• #20: Data Quality-as-a-Service A very interesting option for the future It means that vendors offer DQ Hosted Service Available as a Subscription Service You can sign-up and get some hands-on experience very easily and quickly and FREE (eg SalesForce.com) People are even talking about Data-Governance-as-a-Service DQ plus associated SOPs There is a Data Governance Institute - http://datagovernance.com/ Boomi The newest kid on the block I had a virtual guided tour an iMeeting using a Global Conference facility SalesForce and Business Objects SalesForce and Informatica interesting combinations I worked with SalesForce at Cisco 4 years ago and they are dominant in the SaaS Space I also came across another British company called Kognitio offering a FREE Data Discovery exercise You could get started with this free exercise for Data Profiling, and then sign up for a period of 6 or 12 months. They are based in Bracknell and Marlow. What we are seeing here is the ‘Open’ option where we can use Web Services to link ‘Engines’ together This leads us to “Open Source” Tools (and Talend) …
• #21: Open Source is Cheap to buy and install This option is for people who are prepared to take on more of the Support burden. Professional Support is available on a commercial basis but it stops short of the ‘hand-holding’ provided by the Enterprise products. Talend – have a Chinese Office (and two in California) - They say “ first provider of open source data integration software “ - “150,000 downloads and winning awards” - USP is Open Source, they say it’s DQ-on-Demand but 180Mb download for DQ-a-a-Svc - Looks interesting but 180Mb for download doesn’t seem like DQ-a-a-S - Uses Ingres as a Database, whereas most Open Source use MySQL * Good Blog http://blogs.zdnet.com/BTL/?p=4880 SQL Power - Canadian offers a download option so that you can started without incurring any costs. Gets ticks in many Data Warehouse boxes They offer Dashboard, Data Modelling and Data Profiling – let’s look at that -
• #22: Tool Vendors – SQL Power An excellent example of a Data Profiling analysis It shows the power of the technique Helps with Data Validation and Data Cleaning The Pie-Chart on the right shows the relative frequencies of Product Categories Red upper right ‘Outdoor Soccer’ [181] Blue lower right ‘Indoor Soccer’ [172] In passing, a varchar(30) is not a good Product ID, therefore we should consider having an Enterprise Data Model to provide a foundation But this analysis shows very valuable profiling results
• #23: Top-level Support Board of Directors + Champion Governance SOPs that must be agreed at top-level then rippled down Master Data Management (MDM) A common requirement for Products, Suppliers,Ref Data “A Single View of Things of Interest” CDI Leads to MDM Visible results – everybody understands and describing some duplicates brings the lessons home, such as 17 Bevans. Need to engage with the Business and a Data Platform is very important …
• #24: Platform Priorities - For example, Reference Data, Products, Customers, and so on. EDM, publish, migrate See next slide … Roadmap Vital, need to know where you are going – ie your desired end-state or ‘To-Be’ situation How do we get there from here ? What is our ‘As-Is’ – present % Good Data ? Accountability Who does what – Roles and Responsibilities Stethoscope Monitor key points An organisation is like an organism We need to have a view in order to decide where to use the Stethoscope A Business View of the Data leads us to the need for a Data Platform, So let’s look at that …
• #25: Reference Data underpins the Foundation Properties (Gazetteer) Services (LGSL) Customer (CMI) Customer Services BI Data Mart This Data Platform supports analysis – who takes many services ? Which services are most popular ? Any expensive services not really used eg ambulances available 24x7 but rarely used The Objective of the Data Platform is to establish a foundation for clean quality data Feeding into the Platform is data from various Sources Coming out is unified Data of a Clean Quality. Because you can’t integrate it if it’s not Clean
• #26: Match and Consolidate Customers Eg Joe Bloggs, Joey Bloggs, Joseph Bloggs. Mr J Bloggs and so on. Ealing Film Studios – Alec Guinness who was born Alec Guinness de Cuffe in Marylebone, London, April 1914. His mother was Agnes Cuffe but there is no father's name on his birth certificate. His last name was changed twice before he reached the age of fourteen . For Public Consumption he was called Alec Guinness   Between 1938 and 1941 he played 34 roles in 23 plays – therefore had 34 different names In 1941 he enlisted in the Royal Navy (as Alec Guinness ?) a landing-craft operator In 1951, he starred in “The Man in the White Suit” made at Ealing Studios (but no white hat) On Official documents he would be Alec Guinness Cuffe or Alec Guinness de Cuffe In Ealing, we have many Polish and Somali residents. Different nationalities makes it different to match names of individuals. There are many Arabic and Indian names – where the son can have the same name as the father . and associated date of birth with name is necessary for uniqueness. Sometimes, mothers move house but don’t want to be traced so give a different name, or register children with different names We found 17 versions of John Bevan – when you show it to users it makes a big impact. Need Global ID and CMI – Customer Master Index CDI software Limit to how far you can go with DIY Tools In the diagram, Customers are referred to as Business Owners, Council Tax Residents, HB Claimants, Vehicle Owners and Tenants. Different IDs Therefore, we need the ability to have aliases ‘also known as’ In other words, the software we use has to allow us to define our matching Rules. This is a category of DQ Tools called De-Duping and I have a page on my Web Site listing some products.
• #27: Participants Have input to standards process eg Debtors Data Mart – “What is a Debtor ?” eg Spurs pay about £1 million in Haringey In Ealing we issue Parking Tickets and these represent Debts at some point which has to be defined Performance Reporting – Merge to common Reporting Platform Data Quality Standardisation Layer Supports mapping from many (dirty) sources to one (clean) target An Enterprise Data Model is very important …
• #28: Ealing Data Model The Model is on the Web Site – search for Enterprise Data Model at www.ealing.gov.uk This background shows the impetus to have an EDM created The motivation was the requirement for a consistent approach to Clean Data for Data Marts
• #29: On Ealing Web Site – Search for “Enterprise Data Model” Contact me if interested on email address given on the Web Site or at the end of this Presentation
• #30: This diagram shows clearly the importance of Good Data because you can only consolidate data which matches. “ Apples and Oranges” For example, standards for Customer Addresses requires clean data that can be matched This gives you an idea of how the Model is constructed and the implications for Data Quality Clean and consistent- Addresses, Customers, Services etc. Data which is “Fit for Purpose” for the “Things of Interest” ie “A Single View of the Things of Interest”
• #31: The Data Quality Layer clarifies the role of the Enterprise Data Model Below the DQ Layer there are many Data Sources Above it there is only one (view of the Things of Interest) The DQ Layer includes Mapping and application of Business Rules Let’s turn now to look at Getting Started on the Road to Success …
• #32: Local Authorities are lucky Publish initial Standard values and Set-up Data Governance starting with accepted Procedures for approving changes to Standards.
• #33: These Steps will establish a Strategy for Enterprise Data Quality It’s a substantial undertaking A key phrase is “Data Quality is an Enterprise Issue” It requires support from the Chief Exec It requires commitment over the long haul Success requires Results at regular intervals Starting with “Quick Wins” is a good idea and you should look at easy-to-address problems everybody agrees about We have done this very effectively with our new SMPL Street Management Performance Reporting System
• #34: These are some of the Factors at Ealing which drive a need for good quality data and standardised Reporting They also drove the development of the Enterprise Data Model Customers are a good starting-point They are recorded in many different Systems
• #35: Senior Management Look for a Business Champion Publicly support the DQ Initiative Provide Funding Resolve Issues Remove Roadblocks Line-of-Business Managers Champion the Cause Articulate the DQ Benefits Ensure staff buy-in Data Stewards Drive specific Requirements Provide Feedback Participate in UAT Activities DQ Professionals Maintain Data Dictionary Plan and administer DQ Tasks Implement Business Rules for Cleansing, De-dup’ing Support Data Stewards Run day-to-day DQ operations
• #36: I have been working with a Director at Ealing for the past 18 months who has all these characteristics We have been able to make excellent progress and arrive at a point where we can say “Look at what we have achieved” Without getting bogged down in time-consuming meetings, discussions of approaches and detailed analysis of costings. Now we are building on the foundation to extend the Application to other areas All data has the same characteristics – location, date and time, staff id, observations and follow-up – eg Parking and Abandoned Vehicles Therefore the approach of a Library of Data Quality Validation routines is perfect. The Director had a Vision which included this foundation and I shared the Vision and was able to implement it.
• #37: Agree an Overall Timetable One Year to achieve consistent DQ across the board With SMPL we are achieving this – adding Parking KPIs after one year Three months to obtain buy-in at the working level Quick Wins – what has the Chief Exec asked for and not (easily) been given ? Need a Road-Map to decide “How do we get there (“To-Be”) from Here (“As-Is”)
• #38: Launch with slogan “Data Quality is an Enterprise Issue” Almost any Data Quality work has Enterprise implications eg Gender standards, and Flag_YN with Y/N instead of 0 or 1. Proof-of-Concept Benefits are it’s Hands-On with Deliverables and leaves a Foundation But it addresses a limited and clearly-defined Scope A Feasibility Study, on the other hand, has a broader scope but is theoretical and doesn’t address some very serious issues of involvement and commitment.
• #39: Consider Starting with a DQ Audit Sell it as the first Step in an important Enterprise-level Commitment Aim for a Limited Scope – eg can use SQL Include Profiling to suggest Standards Identify Benefits (Deliverables) – eg Ethnic Origin breakdown or HR Headcount Determine Dependency on people in key positions Obtain buy-in from people affected Data Owners can get defensive It’s like a slice down the organisation Think of Data Quality as using a Stethoscope – Understand the organism, the ranges of the data and thresholds for quality Can get started FREE by asking vendors for trials or advice Eg Send sample files to DQ Now for free Audit
• #40: Thank you for your time I hope you found my Presentation thought-provoking Feel free to email me Comment on my Data Cleansing page Join my Community it’s like a Facebook for Data Management Professionals – to build up Best Practice in key areas What would you like ? 1) A Tutorial based on today’s Presentation with Templates that you could use 2) Vendor hands-on demos 3) an Online Checklist and self-assessment facility to help “As-Is” 4) Strategy for Global Enterprise Data Management ? Good luck with your Data Quality Projects and keep in touch