Shift Money 2019 - Why your data quality problem is not your user's fault - Murat Abur (Suade)
0 likes60 views
The document discusses the critical importance of data quality in financial regulation, emphasizing that bad data—defined by missing attributes, lack of validations, and derived values—can lead to significant issues. It outlines rules and guiding principles for data governance to ensure high-quality data in financial institutions. The piece advocates for clear definitions and avoidance of flags or derived values to maintain data integrity and prevent future financial crises.
Shift Money 2019 - Why your data quality problem is not your user's fault - Murat Abur (Suade)
- 1. DISCOVERING FIRE OR WHY YOUR DATA QUALITY PROBLEM IS NOT THE USER’S FAULT www.suade.org/fire
- 2. HOW DO WE PREVENT THE NEXT FINANCIAL CRISIS? REGTECH...
- 3. R(x1, x2, ..., xn) = y where, y = metrics, reports, analytics, insights R = rules, regulations x = data THE REGULATORY EQUATION
- 4. We think garbage is acceptable Garbage input is not our problem Garbage output also not our problem TECHNOLOGY SAYS GARBAGE IN, GARBAGE OUT Leverage your data model to prevent garbage in!
- 5. Objectives: Quality, control and governance for Financial Institutions Comparability and supervision of underlying data for Regulators Seamless integration for FinTechs and third parties FINANCIAL REGULATION (FIRE) DATA FORMAT Github: https://github.com/suadelabs/fire
- 6. WHAT IS BAD DATA? Leverage your data model to prevent garbage in! Bad data is data with: No definitions No validations Missing values Wrong values Flags Derived values https://xkcd.com/1838/
- 7. BAD DATA: NO DEFINITIONS Amount in USD Total Deposits ? ACT_DEP_BAL R10_USD_AMT GTEE_AMT LIQ14_TOT_BAL R12_AMT 90 60 60 30 120 Rule 1: No attribute without definition Data: Output:
- 8. BAD DATA: NO VALIDATIONS Rule 2: No attribute without limitation Data: CCY START_DT CATEGORY BALANCE USDRATE USD_BAL GBP 2019-03-08 Retail 30 1.20 36 ALL 18/11/20 International -20 0.01 0 USD 18/11/19 Govt 15 100 15
- 9. BAD DATA: MISSING/WRONG VALUES OK. Missing values tell you what you don’t know Data: CCY START_DT CATEGORY BALANCE USDRATE USD_BAL GBP 2019-03-08 Retail 30 1.20 36 ALL International -20 0.01 USD Govt 15 15
- 10. Total USD Deposits from SMEs ? Deposits from Govts ? Deposits from other customers 15 O_REP R10_TYPE CATEGORY USD_AMT SME? GOVT? GB Central Bank Retail 30 True True UK Small Business International 20 False True DE Financial Govt 15 False False BAD DATA: FLAGS Rule 3: Flags are not your friend Output:Data:
- 11. BAD DATA: DERIVED VALUES Amount in EUR Total Deposits ? EURUSD EURGBP GBP_AMT TYPE USD_AMT 1.00 1.16 100 DEP 120 Rule 4: Avoid attributes that can easily be deduced from others
- 12. ADHERE TO PRINCIPLES, NOT YOUR BOSS Good data has: Definitions Validations No flags No derivable values
- 13. FIRE GUIDING PRINCIPLES 1. Data attributes should always be true: “...every pull request requires a corresponding, documented, legal reference, preferably to a currently in-force financial regulation.” 2. Data attributes should be atomic “One property should not be derivable from other properties.“ 3. Data attributes should be consistent “Schema properties should try to avoid logical inconsistencies.”
- 14. TO CONCLUDE Death List OOOO----RENRENRENREN ISHIIISHIIISHIIISHII(added columns without defintions) VERNITAVERNITAVERNITAVERNITA GREENGREENGREENGREEN (added columns without validations) BUDDBUDDBUDDBUDD(added a whole bunch of flags) ELLEELLEELLEELLE DRIVERDRIVERDRIVERDRIVER (added redundant information) BILLBILLBILLBILL (allowed this to happen)