SOLVING DATA RELIABILITY AND VALIDITY PROBLEMS WITH ANALYTICS
- 1. Solving DATA ‘RELIABILITY’ & ‘VALIDITY’ Problems with Analytics BY N ILANGO Assistant Professor, Department of MBA Sri Ramakrishna College of Arts & Science, CBE
- 2. Data Reliability & Data Validity Solving Data Reliability & Data Validity issues ensures: …… Business Decisions are based on……….. Accurate, Consistent & Meaningful …….. Data Understanding the Concepts Data Reliability Refers to consistency of data over time & across systems If same input produces different outputs, data is not reliable Eg: A customer’s name is spelled differently in two databases. Data Validity Refers to accuracy & appropriateness of data for specific purpose If data doesn't measure what it's supposed to, it is not valid Eg: A survey asking for age but receiving zip codes instead
- 3. Solving Data ‘Reliability’ Problems with Analytics A. Inconsistent Data Across Systems Solution: Use data profiling & data integration tools Method: Analyze records across systems (e.g., CRM vs. ERP) to identify mismatches Use tools like ETL (Extract, Transform, Load) to harmonize data formats Eg: Customer order data shows different totals in Sales & Finance systems Run reconciliation analytics to match transactions line-by-line & flag inconsistencies
- 4. Solving Data ‘Reliability’ Problems with Analytics B. Duplicates or Redundant Entries Solution: Use de-duplication algorithms & fuzzy matching Method: Use rules like “Same phone number + similar name” to identify likely duplicates Analytics software (e.g., Power BI, Tableau Prep, Talend) can visualize clusters Eg: Two entries: “Mohan Babu” & “M. Babu” with same phone number Flag & merge records using BI dashboard
- 5. Solving Data ‘Reliability’ Problems with Analytics C. Time-based Inconsistencies Solution: Perform time-series analysis Method: Look for unexpected data spikes, gaps or drops using trend lines Eg: Daily website traffic drops to zero suddenly — analytics helps detect missing data imports
- 6. Solving Data ‘Validity’ Problems with Analytics I. Incorrect Data Types or Ranges Solution: Apply validation rules & data audits Method: Check if “age” is within realistic bounds (e.g., 18–99) Use dashboards to highlight values outside normal ranges Eg: If a customer’s age is recorded as 543, then, Analytics dashboard flags it as a red alert
- 7. Solving Data ‘Validity’ Problems with Analytics II. Missing or Null Values Solution: Use null analysis & imputation models How: Highlight % of missing data in each field Use predictive models to fill gaps (e.g., estimate missing income based on location & job) Eg: If, 30% of "Annual Income" values are left as blank, then, Analytics fills in values using regression models
- 8. Solving Data ‘Validity’ Problems with Analytics III. Out-of-context Responses Solution: Use text analytics & semantic validation Method: If a survey asks for a city & the response is "Great!", NLP can detect irrelevance Eg: Text analytics tags invalid responses & routes them for cleanup or exclusion
- 9. END