Nexx Consultants: Credit risk/IRB Approach models validation
3 likes1,574 views
Nexx Consultants, a Toronto-based financial services consulting firm, specializes in improving risk models, processes, and reporting for banks and financial institutions. The firm employs a proprietary validation tool to conduct model validation tests and offers a range of services, including credit risk modeling, training, and customized consulting. Their approach focuses on enhancing client capabilities through collaborative partnerships while ensuring compliance with regulatory standards.
Nexx Consultants: Credit risk/IRB Approach models validation
- 1. Copyright©2016NexxConsultants Disclaimer Nexx Consultantsis incorporatedunder theLaws of Provinceof Ontarioin Canada, Corporation No: 002369533. The information,dataand approaches provided hereinare anoutcome ofour researchandcontent expertise.Althoughwe may refer tothirdparty materials and/or analyze their impact, the contentis theoutcomeof Nexx's proprietaryknowledgeand is rightfully owned by us.This work is copyright protectedandlegally privileged. Please donot distributethis presentationwithout theprior writtenconsent of Nexx or its authorized affiliates. Nexx has madebestefforts to ensurethat this material is complete in its entirety. However, wedo notwarrantits completeness,accuracy,usefulness or satisfaction with all requirements. Copyright © 2016 Nexx Consultants Select assistance themes: Credit risk/IRB Approach Models validation Contact us: sfarooq@nexxconsultants.com April/2016
- 2. Copyright©2016NexxConsultants Content 2 Introduction to Nexx Consultants2 Our models validation philosophy Executive summary1 3 4 Overview of models validation approaches 5 Appendices Appendix 1: Case studies Appendix 2: Screen shots of our validation tool
- 3. Copyright©2016NexxConsultants Content 3 Introduction to Nexx Consultants2 Our models validation philosophy Executive summary1 3 4 Overview of models validation approaches 5 Appendices Appendix 1: Case studies Appendix 2: Screen shots of our validation tool
- 4. Copyright©2016NexxConsultants Executive summary 4 • Nexx is a Toronto-based FS consulting firm, assisting banks and FI's improve their risk models, processes and reporting • Nexx offers a pool of highly specialized consultants, with prior experience at top-tier consulting firms, such as, Oliver Wyman, Roland Berger and BCG. Our consultants have completed several projects for a range of clients, including, Systematically Important Financial Institutions covering topics, such as, development of risk models, validation and stress testing • Nexx uses proprietary validation tool to conduct validation tests. This tool has been designed specifically to undertake validation tests. In addition, models developed by our consultants have been independently vetted, validated and accredited by global regulators • This presentation provides a brief introduction to the firm and offers high level insight into our models validation content expertise • In addition to the services offered herein, Nexx also provides: o Factor-level benchmarks, including factor weights o Dual calibration of PDs o Development of master rating scale o Risk engines parameterization o Risk-based pricing o Customized training on models design, development and validation o Credit portfolio management o Credit process re-design o Limit setting and its integration with risk appetite • Our deliverables include presentations on models performance, spreadsheets used in model tests, analysis and documents prepared for regulatory reporting
- 5. Copyright©2016NexxConsultants Assistance theme to serve a cross-section of stakeholders Stakeholders Assistance themes Estimated effort ChiefExecutive Officer • Upfront identification ofvolatile portfolios (due to higher provisions) and approaches to address volatility,including: o Creditportfolio strategy o RWA optimization • Balance sheetrepositioning • Risk-based pricing • 12 FTE months • 12 FTE months • 60 FTE months • 12 FTE months • 8 FTE months ChiefRisk Officer • Dual calibration ofPDs • Data and system design and architecture • Increased portfolio granularity and changes in master rating scale • Internal reporting • Collections and recovery • 16 FTE months • 24 FTE months • 12 FTE months • 6 FTE months • 12 FTE months ChiefFinancial Officer • IFRS 9 modelling and validation • Rating agency/stakeholder reporting • Draft commentary on provisioning • 60 FTE months • 6 FTE months • 3 FTE months ChiefCompliance Officer • Governance and policy design • 8 FTE months Internal Auditand Assurance • Review and verification of approaches • References and benchmarking • 3 FTE months • 3 FTE months 5
- 6. Copyright©2016NexxConsultants Content 6 Introduction to Nexx Consultants2 Our models validation philosophy Executive summary1 3 4 Overview of models validation approaches 5 Appendices Appendix 1: Case studies Appendix 2: Screen shots of our validation tool
- 7. Copyright©2016NexxConsultants We are an independent FS risk consulting firm offering, risk and inter-domain expertise Risk Domain Risk • Re-positioning the risk function from compliance center to adding shareholder value • Risk measurement and capitalization • Strategy Implementation • Program Management • Internal consultant for on-demand training • Compliance and governance framework design • Discovery of value drivers • Policy development • Delegated authority framework • Crisis and event response, including first responders, etc. • Data aggregation • Business and IT alignment • System and risk applications alignment • Capabilities and assessment review Treasury and Finance • Integrated planning • Capital management • Balance sheet mixing • Behaviour analysis of deposits 7
- 8. Copyright©2016NexxConsultants Credit risk Credit rating Scorecards development across various asset classes Loss parameters estimation PD, LGD, EAD models development, calibration and master scale development Validation Powerstats, K-S, Gini, AUC Pricing Risk adjusted return on capital (transaction and portfolio) Credit portfolio Loss distribution/parameter estimation/Credit Portfolio Management Provisioning IFRS 9, Dynamic provisioning fund Market risk Securities Portfolio optimization, Performance attribution, Expected Shortfall and VaR Financial instruments Valuation, development and implementation of interest rate libraries Liquidity risk ALM Interest rate risk in banking book Cost of funds Transfer pricing/yield curve development Balance sheet Behavioral risk analysis of deposits Liquidity risk Liquidity survival horizon Operational risk Measurement Loss distribution, scenario-based capitalization of operational risk Management Risk Control Self-Assessment (RCSA), Key Risk Indicators’ attribution Non-financial risks Value drivers Risk appetite design and implementation Capital management Planning and budgeting Integrated planning, forecasting Regulatory Pillar 1, Pillar 2, Capital buffer management Economic capital Stress testing, Economic capital modelling An illustration of our domain capabilities 8References available1 1
- 9. Copyright©2016NexxConsultants Our engagements approach: Our engagements are designed to upskill client capabilities 9 Low High Clients with limited internal capability Clients with high internal capability Specialist engagement approach mapped to client’s capabilities Our collaborative approach allows division of labour based on specialization CIO/CTO/CDO CRO CFO CEO Staffing solution to fit all project sizes Project sponsors Resource availability at client site
- 10. Copyright©2016NexxConsultants 10 In a nutshell….... Creative engagement model We offer three permutations of engagement model: • Content • Consulting • Training Strong content expertise: Risk and inter- domain expertise We are risk experts. We focus on risk and its intersection with other domains. We assist clients improve the models, tools, and processes to manage risk Delivery track record Nexx Consultants are experts in the delivery of large scale and complex risk programmes. Our deliverables have been independently validated and accredited by regulators Knowledge transfer Our engagements are designed to upskill client capabilities. Our collaborative approach allows division of labor based on specialist model Consulting team experience Each member of our team brings a unique perspective - backed by an average of 10 years relevant industry experience gained at the world's leading Financial Institutions and consulting firms We deliver solutions tailored to our clients profiles Why Nexx Consultants?
- 11. Copyright©2016NexxConsultants 4Sohail Farooq, Co-founder 418+ years as risk practitioner/consultant, ex-Oliver Wyman, ex- Moody’s 4Specialties: Risk measurement, validation, capitalization and stress testing 4Education: MQF, MBA Nexx’s senior team 11 4Dr. Michel Crouhy, Co-founder and Subject-matter expert 425+ years as risk practitioner 4Specialties: Validation, stress testing, including CCAR, DFAST 4Education: PhD Finance, Doctoris Honoris Causa 4Dietrich Matthes, Cooperation Partner 420+ years as risk practitioner/consultant, ex-Oliver Wyman, ex- Roland Berger, ex-KDB 4Specialties: Models development, validation, stress testing and provisioning 4Education: PhD Quantum Field Theory
- 12. Copyright©2016NexxConsultants Benefits of engaging Nexx 12 Continuity of expertise,not of brand Knowledge transfer and upskilling • Our niche focus is built upon our pool of highly specialized staff. This ensures that: o There is true client-consultant continuity at a resource level o We establish a thorough understanding of the specific business needs, as well as the unique market, organizational and economic reality each client faces o This understanding is maintained throughout a given project and subsequent to the conclusion of our work because we continue to provide advice and perspectives on certain topics and market developments to our clients Fixed price business model • Knowledge transfer is fundamental to the success of our projects • We make knowledge transfer “business as usual” so that we can deliver tangible and sustained business value • Individual engagements are designed on the basis of o “Appropriateness of fit” within the organization o Building capabilities within our clients that outlast Nexx’s involvement on the project • Our teams work closely with client counterparts to promote the exchange of knowledge • Our business model typically operates on a fixed price contract basis, which ensures that there is budget certainty without extension risk to the client • We truly partner with our clients to share the project risks, adopting a “we are in the same boat” mentality
- 13. Copyright©2016NexxConsultants The team has completed projects across a wide cross-section of clients – some examples *Anonymized project details can be made available 13
- 14. Copyright©2016NexxConsultants Content 14 Introduction to Nexx Consultants2 Our models validation philosophy Executive summary1 3 4 Overview of models validation approaches 5 Appendices Appendix 1: Case studies Appendix 2: Screen shots of our validation tool
- 15. Copyright©2016NexxConsultants Nexx offers full range of coverage across major asset classes on models development, refinement, validation and benchmarking Core non-retail portfolios Real estate Financial institutions Large Corporate Small Business Real Estate Development Banks NBFI Segment 1 Real Estate Investment Retail portfolio SME (Operating) Holding Company Specialised segments Real Estate Portfolio Project Finance Share Financing Consumer Product A Potentially limited default data Consumer Product B … Sovereign and PSE Sovereign PSE Others depending on portfolio size, e.g. Non-profit, High net-worth individuals, Startups, etc.Construction Non-domestic corporates NBFI Segment 2 … Recently restructured Middle Market 15
- 16. Copyright©2016NexxConsultants Our experience also includes development and vetting of retail scorecards • Origination decision • Consistent enforcement of underwriting standards • Underwriting cut-off • Up-front campaign planning • Economic analysis of new originations • First step to life-time view • Link to pricing • (Economic capital and Basel II IRB - stabilised) Application Scorecards development objectives Behavioural Scorecards development objectives n Portfolio assessment – Risk assessment for loans on the book – Loss forecasting – (Economic capital and Basel II IRB) n Customer level decisions – which customers are worth investing in them – Limit management – Activation campaigns – Retention management – Cross sell to other products n Collections – Tiering of collection efforts 16
- 17. Copyright©2016NexxConsultants We assist with validation at varying stages of model’s life cycle Higher performance Higher performance Lower performanceTime of rating tool construction / revision Existing portfolio New portfolio Time Validation during development In-sample Out-of-sample Used for build the model during the development phase Out-of-time (ongoing validation / backtesting) Out-of-universe In-universe Shows how well model works on new customers Shows how well model works on existing customersUsed for test the model during the development phase Development sample Validation sample This can only really be done once a model is in use Lower performance 17
- 18. Copyright©2016NexxConsultants Our validation philosophy is underpinned in best practice principles of validation Quantitative validation n Statistical proof of rating model performance n Objective tests for senior management and regulatory review n Core part of validation (where possible) Qualitative validation n Acceptance and credibility with the management n Proof of pervasive use within the organisation n Development approach in line with standard or best practice (over portfolio) Principles of validation 1. Validation fundamentally about the predictive ability of rating models and use in credit processes 2. The financial organization has primary responsibility for validation 3. Validation is an iterative process 4. There is no single validation method 5. Validation should encompass both quantitative and qualitative methods 6. Validation process and outcomes should be subject to independent review 18
- 19. Copyright©2016NexxConsultants Three dimensions of quantitative validation Implication for PD model validation Discriminatory Power (Rank ordering, or separation ability) Accuracy (Calibration) ‘Stability’ n Ability to discriminate healthy borrowers from troubled borrowers (usually taken as ability to discriminate defaults from non-default) Typical Tests What is measured? n Range of accepted statistical tests e.g. (ROC, AR, Gini, K-S), based on sample of goods and bads n Other tests are possible when externalindicators of credit quality are available, e.g. externalagency rating, internal expert ratings – ‘Implied accuracyratio’ – Rank correlation or matching matrices n Ability to assign accurate ‘long-term’PDs to each rating (i.e. to obligors in each rating band) n Comparison of actualvs. predicted PD – is performance with expected error bounds? n Other tests are possible, for example: – Comparison to PDs implied by externalagency ratings – ‘High-level’ cross comparison across portfolios to assess accuracyof relative rating profile n Stable, causal relationships between factors and credit quality over time n Statisticaltests would rely on the ability to monitor model performance over time, and ensure model continues to show discriminatory power in line with expectation, and that model factors continue to be significant and predictive Implication for validating an expert-based FI model n Regulators will expect to see some validation/comparison of ranking ability (e.g. versus agency or internal expert ratings) n At the very least, a high level comparison acrossportfolios should be reviewed to get an expert assessment that absolute PDs reasonable acrossand within segments n Need to demonstrate of consistencyof rating assignment 19
- 20. Copyright©2016NexxConsultants For quantitative validation we focus on three core dimensions Discriminatory Power (Rank ordering, or separation ability) Accuracy (Calibration) ‘Stability’ n Ability to discriminate healthy borrowers from troubled borrowers (usually taken as ability to discriminate defaults from non-default) What is measured? n Ability to assign accurate ‘long-term’ PDs to each rating (i.e. to obligors in each rating band) n Stable, causal relationships between factors and credit quality over time n Overall, the most important test: – For banks: Accurate PDs for the ‘use test’ – For regulators: Accurate RWA parameters 20
- 21. Copyright©2016NexxConsultants Content 21 Introduction to Nexx Consultants2 Our models validation philosophy Executive summary1 3 4 Overview of models validation approaches 5 Appendices Appendix 1: Case studies Appendix 2: Screen shots of our validation tool
- 22. Copyright©2016NexxConsultants Analysis at each ‘level’ of a model should be part of a bank’s ongoing backtesting – however, to avoid excessive analysis, banks can employ a hierarchy of tests First Level 2nd Level 3rd Level Qualitativeassessment Model discriminatory power Segmentation Calibration curve shape GranularityAnchor point Calibration Module 2Module 1 Module 3 Factor 1 Factor 2 Factor 3 … Model validation Accept Reject Modify Accept Accept Accept AcceptAccept Modified Modified Reject 22
- 23. Copyright©2016NexxConsultants For discriminatory power, the Accuracy Ratio is commonly accepted as the best statistical measure. . . the ‘first level’ test would test accuracy for the complete model 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Perfect Model Tested Model Random Model B A Cumulative Bads Cumulative Sample (Rank Order) Accuracy Ratio = A A+B Range from 0 – 100% n Other tests include the K-S statistic, CIER, etc n All these tests, especially AR and K-S, are highly correlated n Most regulators stress the need to use multiple tests, but we would emphasise it is reasonable for a bank to have one main test, which is typically the accuracy ration Accuracy Ratio 23
- 24. Copyright©2016NexxConsultants ‘Second level’ testing of PD model calibration (accuracy) relies on error bounds per rating grade and at the overall portfolio level • Is the overall level correctly predicted (previous slide)? • Observed vs. predicted • Impact of cycle/imperfect cyclicality • Shape – are relative levels across classes of PD or ln(G/B) correct? • Correcting for level • Granularity – does the tool maintain a good spread of exposures across classes? • Typically, only if several rating buckets violate their error bounds should a bank consider recalibration • Keep in mind that the data per bucket is often extremely sparse, and that ensuring aggregate PD is accurate is the first order test to be applied Example – Shape test Three tests used in validating model ‘calibration’ Key : l Actual default rates Predicted default rates Volume of accounts PD ‘grade’ -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 log(DefaultRate) 0 1,000 2,000 3,000 4,000 5,000 6,000 NumberofObservations 24
- 25. Copyright©2016NexxConsultants Validation of LGD and EAD models is much less well-established body of knowledge, as compared to PD validation • There are virtually no external publications establishing a standard body of knowledge or approaches to LGD and EAD validation • In the past, most banks did not systematically monitor or validate their LGD and EAD models • PD viewed as bigger driver, and area where more ‘science’ developed • PD easier to validate (since outcome is binary, data easier to capture) • Thus, we expect that significant advances will come over the next several years, some of which may supplant the suggested tests we discuss here 25
- 26. Copyright©2016NexxConsultants The same three overarching dimensions also apply to LGD and EAD Discriminatory Power (Rank Ordering, or Separation Ability) Accuracy (Calibration) ‘Stability’ n Ability to discriminate high EAD/LGD facilities from low EAD/LGD facilities Typical Tests What is measured? n Modification of traditional accuracy ratio or similar measures n Ability to assign accurate ‘long- term’ PDs to each rating (i.e. to obligors in each rating band) n Overall bucketing of actual versus predicted LGD/EAD n Mean-squared error (MSE) tests to tests overall dispersion around estimates n Stable, causal relationships between model factors and LGD/EAD over time n Stability of performance tests over time n Stability of underlying model parameters (e.g. collateral recovery rates) 26
- 27. Copyright©2016NexxConsultants Content 27 Introduction to Nexx Consultants2 Our models validation philosophy Executive summary1 3 4 Overview of models validation approaches 5 Appendices Appendix 1: Case studies Appendix 2: Screen shots of our validation tool
- 28. Copyright©2016NexxConsultants For model accuracy, the key test is to compare actual versus predicted LGD or EAD, to identify if results are within acceptable statistical error bounds Asian client example: LGD validation – Accuracy test Very small sample – results statistically inconclusive Overly conservative unsecured recovery rate led to overestimation of LGD for highest risk facilities Banks can also apply an MSE test of dispersion at the facility level, which is harder to depict graphically LGD Validation 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Bucket 1 (0% - 12.5%) Bucket 2 (12.5% - 25%) Bucket 3 (25% - 37.5%) Bucket 4 (37.5% - 50%) Bucket 5 (50% - 62.5%) Bucket 6 (62.5% - 75%) Bucket 7 (75% - 87.5%) Bucket 8 (87.5% - 100%) LGD 28
- 29. Copyright©2016NexxConsultants When the test result is below expectation, the more detailed analysis should help isolate drivers of the problem 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% Percentile % Bad Naive Perfect Tested Owner sub-model power curve 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% Percentile % Bad Naive Perfect Tested 0% 20% 40% 60% 80% 100% 0% 20% 40% 60% 80% 100% Percentile % Bad Naive Perfect Tested Financial sub- model power curve Non-financial sub-model power curve Case study: Small business model validation n Predictive power of small business model dropping to ~40%, at a time when other banks were building better models n Validation by sub-model revealed that weight (30%) given to non-financial (largely subjective) model was destroying power n Results: Most non-financial factors expunged from model, and overall model re-weighted toward owner/behavioural factors AR ~ 40% AR ~ 18% AR ~ 6% 29
- 30. Copyright©2016NexxConsultants Example: Credit Cards EAD analysis – Different customer profiles exhibit different race to default behaviour … Revolver n Significant increase in limit leading up to default n Balance often increases beyond limit at default due to accumulated interest Transactor n Increase in limit and balance leading to default n Utilisation increases from low average to a normal high limit n Depends strongly on the type of limit management 0 20 40 60 80 100 120 140 160 NT$'000 Dormant RevolverTransactor Balance Limit Balance Limit Balance Limit At DefaultInitial At DefaultInitial At DefaultInitial Dormant n Race to default most significant for dormant accounts n Utilisation increases from an average close to 0 to a normal high utilisation … with bottom-up vs top-down reconciliation necessary – eventually size and PD segmentation drivers have to be tested to avoid failure of reconciliation 30
- 31. Copyright©2016NexxConsultants Example: Potential outcomes for testing overall calibration accuracy (‘anchor point’) Potential outcomes for the ‘anchor point’ test of calibration n Note that unless the validation period under evaluation is very long or the acceptable default rate band is very narrow, the result will be ‘ambiguous’ most of the time – this is simply the nature of PD validation, and the reason that tests must also be augmented with expert review and judgment n Normally, a bank should consider a conservatism upward adjustment of PDs is the tests are showing ‘ambiguous negatives’with actual default rates below the anchor band Observed Default Rate Confidence Interval Validation failed Validation achieved Validation ambiguous, negative indication Validation ambiguous, positive indication Minimum expected Default Rate Maximum expected Default Rate Observed Default Rate Confidence Interval Validation failed Validation achieved Validation ambiguous, negative indication Validation ambiguous, positive indication Minimum expected Default Rate Maximum expected Default Rate 31
- 32. Copyright©2016NexxConsultants Content 32 Introduction to Nexx Consultants2 Our models validation philosophy Executive summary1 3 4 Overview of models validation approaches 5 Appendices Appendix 1: Case studies Appendix 2: Screen shots of our validation tool
- 34. Copyright©2016NexxConsultants Detailed analysis at model, module or single factor level 34
- 38. Copyright©2016NexxConsultants Calculation of key statistics 38
- 39. Copyright©2016NexxConsultants Disclaimer Nexx Consultantsis incorporatedunder theLaws of Provinceof Ontarioin Canada, Corporation No: 002369533. The information,dataand approaches provided hereinare anoutcome ofour researchandcontent expertise.Althoughwe may refer tothirdparty materials and/or analyze their impact, the contentis theoutcomeof Nexx's proprietaryknowledgeand is rightfully owned by us.This work is copyright protectedandlegally privileged. Please donot distributethis presentationwithout theprior writtenconsent of Nexx or its authorized affiliates. Nexx has madebestefforts to ensurethat this material is complete in its entirety. However, wedo notwarrantits completeness,accuracy,usefulness or satisfaction with all requirements. Copyright © 2016 Nexx Consultants Thank You