Independent models validation and automation
1 like290 views
The document presents a comprehensive overview of independent third-party validation and automation of validation tests for risk and capital models in the financial sector. It emphasizes the importance of ongoing model monitoring as mandated by regulators, detailing various validation tests and governance frameworks to ensure accuracy and compliance. Additionally, it includes case studies and methodologies for improving model validations while optimizing risk management processes.
Independent models validation and automation
- 1. Copyright©2018BBAnalytics Copyright © 2018 BB Analytics Independent validation and its automation
- 2. Copyright©2018BBAnalytics Contents Executive Summary1 3 Independent 3rd party validation Automation of validation for 2nd and 3rd lines of defense 2 2
- 3. Copyright©2018BBAnalytics Contents Executive Summary1 3 Independent 3rd party validation Automation of validation for 2nd and 3rd lines of defense 2 3
- 4. Copyright©2018BBAnalytics Executive summary • Independent third party validation of risk and capital models • Automation of validation tests for 2nd and 3rd lines of defense • The rest of this deck is organized as follows: o Independent validation and description of validation tests and reporting o Automation of validation tests and a case study on cost savings 4 Within the context of validation, BankingBook Analytics (BBA) assists clients with two broad themes:
- 5. Copyright©2018BBAnalytics Contents Executive Summary1 3 Independent 3rd party validation Automation of validation for 2nd and 3rd lines of defense 2 5
- 6. Copyright©2018BBAnalytics • SR 11-7: Guidance on Model Risk Management • Enterprise-Wide Model Risk Management for Deposit-Taking Institutions What is model’s validation and why is it important Regulators require on-going/annual monitoring of material models 6
- 7. Copyright©2018BBAnalytics Regulatory focus is on the following three types of test cohorts… Discriminatory Power (Rank ordering, or separation ability) Accuracy (Calibration) ‘Stability’ n Ability to discriminate healthy borrowers from troubled borrowers (usually taken as ability to discriminate defaults from non-default) Regulatory oversight n Ability to assign accurate ‘long-term’ PDs to each rating (i.e. to obligors in each rating band) n Stable, causal relationships between factors and credit quality over time n Overall, the most important test: – For banks: Accurate PDs for the ‘use test’ – For regulators: Accurate RWA parameters 7
- 8. Copyright©2018BBAnalytics We mapped our tests to each regulatory requirement 8 Discriminatory power Accuracy Stability Probability of default • Rank ordering performance measure (ROPM) tool level (here we would provide a separate ROPM test suite with a complete menu of ROPM measures) • ROPM module level • ROPM factor level • Concentration Analysis (Herfindahl Index) • Stability Analysis (Population stability Index, Chi- squared test) • Relative Risk Level Check • Intra-segment analysis • Inter-segment analysis • Model weight review • The anchor point test • Calibration curve shape test • Granularity check • Bootstrapping Loss Given Default / Exposure at Default • Lorenz Power Curve • Power Stats • Spearman's Rank Correlation • MSE • Bucket analysis
- 9. Copyright©2018BBAnalytics We use a hierarchy of tests to administer model, module and factor level tests Validation tests create escalation only if the results are amber or red First Level 2nd Level 3rd Level Qualitativeassessment Model discriminatory power Segmentation Calibration curve shape GranularityAnchor point Calibration Module 2Module 1 Module 3 Factor 1 Factor 2 Factor 3 … Model validation Accept Reject Modify Accept Accept Accept AcceptAccept Modified Modified Reject 9
- 10. Copyright©2018BBAnalytics In addition to model risk management, BBA also assists with models’ governance, improving accuracy and optimizing risk- measured returns Models governance framework § Maintain a model inventory with portfolio/model characteristics, performance levels and materiality (HO, Group, Division or BU level models) § Understanding models ownership and governance process § Policy and accountabilities for making changes to: model methodology; algorithm, tool, and parameter selection; validation etc. § Set the group-wide validation policy and guidelines to ensure consistency and comprehensiveness § Determine and maintain performance requirements for each model § Set and refine guidelines on an ongoing basis through experience and continuous interaction with users § Determine validation schedule considering materiality and performance level of each model § Interact with users/BU’s to agree on validation schedule § Determine the rigor of the validation considering materiality and performance level of each model § Validate rating systems (and provide final judgement) § Re-validate newly developed rating systems/models (initial validation) § Provide documentation for all group-wide rating systems § Prepare documentation for approval of rating systems § Establish/review the activity plans of the developers and set appropriate checkpoints/ validation timelines § Follow-up on recommendations / review the implementation § Defend validations in front of the regulator § Act as a centre of excellence for model build across the organization – Benchmark model design and performance across similar portfolios – Interact with model developers throughout the development process to ensure compliance with validation standards § Provide recommendations for model improvement and ensure developers comply with the agreed timelines Designing the validation schedule and rigor Improving and optimizing portfolio of models Identifying benchmarks and trigger levels to meet minimum requirements Key focus items 10
- 11. Copyright©2018BBAnalytics Case study: Rank ordering test reporting 11 Perfect model – Maximum predictive power Bads . . . . . . Goods Bads . . . . . . Goods Bads . . . . . . Goods Model score Achievable rating model Random model – No predictive power Model score 0 20 40 60 80 100 0 20 40 60 80 100 Model score 0 20 40 60 80 100 Bad Good Accuracy ratios = Area A/Area (A + B) Cumulative % of defaults Cumulative % of total sample Perfect model Achievable model Random model (No differentiation) 0% 100% Area A Area B Best scoresWorst scores Area A Goods in this direction Bads in this direction
- 12. Copyright©2018BBAnalytics Case study: Model risk reporting dashboard and its design Approval of new risk models and material changes to existing risk models1 n Individual accountability is at the core of the proposed procedures for approval of new risk models and material changes to existing risk models – Individuals accountable may wish to base their approval decisions on the work of fora or committees, or seek other kinds of support, but they are ‘on the hook’ for making these decisions n In line with this institution’s risk organization’s structure (Federated), the proposed model approval accountabilities are decentralised 1. A material change is a change that has not been specified in the model documentation as routine. When models are approved, the primary model owner is at the same time given authority to perform routine changes to the model, and she/he is only obliged to seek approval for changes that are not routine, or if a routine change results in significant changes to the output of the model 2. The Chief Manager, Financial Management Information in Group Finance receives notes on all risk-model approval decisions because Group Finance are involved in or have an interest in almost all the risk modelling in the Group. Moreover, Group Finance, which houses Investor Relations, will take a greater interest in any model changes once the risk numbers start to be published 3. The AMSR Director in Group Risk is regarded as the relevant Group Risk Director for all capital models Model Ownership Absolute Model Impact from a Group Perspective Head Developer / User Head of Function (where model is) Head of Risk Managing Director Head Developer / User Head of Function (where model is) Divisional Risk Officer Group Executive Director Head Developer / User Head of Function (if not Head of RM) Chief Mgr, FMI, Grp Finance 2 Head of Risk Modelling AMSR Director 3 Group Risk Director Chief Risk Director Group Chief Executive Risk Oversight C'ttee High R C C C C A I C I C I I I Medium R C C A I I I C I I I Low R A I I I I C I I High R C C C I C I A I I I Medium R C C A I C I I I Low R A I I I C I I High R C I C I A I I I Medium R C I C I A I Low R C I A I I R = Responsible (primary model owner) C = Consulted (makes or plays an active part in making a recommendation regarding the model/change) A = Accountable (person responsible for approving the model as well as material changes to the model) I = Informed (noted) Division Group Division Group BU BU 12
- 13. Copyright©2018BBAnalytics Contents Executive Summary1 3 Independent 3rd party validation Automation of validation for 2nd and 3rd lines of defense 2 13
- 14. Copyright©2018BBAnalytics BBA can also help develop suite of validations tests and automate quarterly validation at FIs FACTOR 2 STANDARDS Insuff. Practice Good Practice Best Practice Specific criteria describing insufficient practice Specific criteria describing good practice Specific criteria describing industry best practice BALANCED SCORECARD Assessment (Illustrative) Data Factor 1 2 x Factor 2 1 x Factor 3 4 x Grading / Calibration Factor 1 4 x Factor 2 4 x Factor 3 3 x Rating Processes Factor 1 4 x Factor 2 2 x Factor 3 3 x Oversight and Control Mechanisms Factor 1 1 x Factor 2 3 x Factor 3 2 x Weight (Illustrative) Qualitative Validation Quantitative Validation n Using our extensive database of factors benchmarks, we would be able to augment standards determined based on benchmarking global/regional banks RATING SYSTEM QUALIFICATION n Does not qualify for IRB approach n Use standard approach n Qualifies for IRB n High conservatism factors should be applied n Qualifies for IRB n Moderate conservatism factors should be applied n Qualifies for IRB n Low conservatism factors should be applied n Qualifies for IRB n Risk weights should be applied as prescribed by BIS 2 Illustrative 14
- 15. Copyright©2018BBAnalytics Current situation at Canadian FIs Understanding models coverage § Credit as a significant activity § Measurement of risk using models § Materiality definition Designing tests Monitoring Automation of validation Models Inventory management Current situation § Insufficient data feed § Ad-hoc analytics § Reporting lacks automation Next steps that we can assist with § What tests are considered acceptable by regulators § Design and conduct of tests § Documentation • Large cross-section of FTEs involved in myriad of tasks, pre-dominant being models validation 15
- 16. Copyright©2018BBAnalytics Our validation application ingests inbound data and provides results by populating reports 17 Data input Analytics and reporting engine Output Flat files/CSV API Batch processAd-hoc Test results Documentation Automatic escalation Administration of tests for PDs, LGDs and EaDs Monitoring Escalations Automated templates to be populated with results Dashboard, etc. Submission Internalaudit interface BBA developed
- 17. Copyright©2018BBAnalytics Typical project workplan 3 – 4 months timeline 18 Checklist and Automation • Model design (c. 28 – 30 tests • Model performance (c. 30 – 35 tests) • Model use-test (c. 30 – 40 tests) • Reporting automation Work-block 1: Current state assessment (1.5 weeks) Work-block 2: Design blueprint (1.5 weeks) Work-block 3: Production (9 weeks) Tool Development • PD, LGD, EAD validation/testing tools in Excel-VBA • Methodology documents • User-manuals Value Capture 0.0 0.5 1.0 1.5 2 3 5 6 7 8 9 10 11 12 Week Number Piloting & Handover • Piloting a test sample of retail portfolio • Handover of deliverables • Knowledge transfer Illustrative
- 18. Thank you for your time! BB Analytics Leading on Solutions | Leading on Impact The Exchange Tower, 130 King Street West, Suite 1800 Toronto, Ontario M5X 1E3 Canada +1-905-499-3618 contact@bankingbookanalytics.com Bankingbookanalytics.com Thank you for your time! BB Analytics Leading on Solutions | Leading on Impact The Exchange Tower, 130 King Street West, Suite 1800 Toronto, Ontario M5X 1E3 Canada +1-905-499-3618 contact@bankingbookanalytics.com Bankingbookanalytics.com