Model Governance and Validation: Best Practices and Common Pitfalls
- 1. © Oliver Wyman MODEL GOVERNANCE AND VALIDATION: BEST PRACTICES AND COMMON PITFALLS MAY 13, 2019 Mark Spong, FSA, CERA, MAAA Simon Li, ASA
- 2. CONFIDENTIALITY Our clients’ industries are extremely competitive, and the maintenance of confidentiality with respect to our clients’ plans and data is critical. Oliver Wyman rigorously applies internal confidentiality practices to protect the confidentiality of all client information. Similarly, our industry is very competitive. We view our approaches and insights as proprietary and therefore look to our clients to protect our interests in our proposals, presentations, methodologies and analytical techniques. Under no circumstances should this material be shared with any third party without the prior written consent of Oliver Wyman. © Oliver Wyman
- 3. 3© Oliver Wyman Agenda 1 5 minutes Agenda and introduction 2 15 minutes Model validation • Pitfalls in a typical project, tools and framework • Distribution of industry findings 3 20 minutes Model governance • Pitfalls across the industry and general application • Observations from an auditor 4 10 minutes Case study 5 10 minutes Buffer and Q&A • Discussion questions 60 minutes
- 4. 4© Oliver Wyman Model validation • A set of processes verifying that models are performing as expected, in line with their design objectives and business uses1 Three dimensions of model risk management Model development & use • Model development, testing, and usage Model governance • A framework with defined roles and responsibilities for model development, usage, communications, and approval 1 Source: SR Letter 11-7 – Supervisory Guidance on Model Risk Management
- 6. 6© Oliver Wyman Typical timeline for a model validation project Month Product feature/assumption validation Data collection Test sample validation Test sample selection Model output review and reconciliation Validation report and documentation of findings 1 2 3 4 5 6 Input validation Calculation validation Output validation Upstream and downstream processes Documentation Status meetings and monitoring Management and oversight Management updates
- 7. 7© Oliver Wyman Common model validation techniques High risk models • Full reconciliation against input source • Assumption benchmarking • Independent full model replication • Independent sample recalculations • Static validation • Dynamic validation • Handoff testing • Backtesting • Implied rate checks • Reconciliation to ledger • Trend analysis • Sensitivity analysis • Rollforward analysis Medium risk models Low risk models • Spot checking • Process approximation • Formula inspection • Static validation • Dynamic validation • Implied rate checks INPUT VALIDATION OUTPUT VALIDATION CALCULATION VALIDATION
- 8. 8© Oliver Wyman Sample model validation framework Model validation is an ongoing process 1 2 3 4 5 6 7 Model identification 1 Model risk assessment 2 Data collection 3 Testing and quantifications 4 Documentation and communication 5 Remediation 6 Sign-off 7 Core validation activities
- 9. 9© Oliver Wyman Sample model validation framework Model validation is an ongoing process 1 2 3 4 5 6 7 Model identification 1 Model risk assessment 2 Data collection 3 Testing and quantifications 4 Documentation and communication 5 Remediation 6 Sign-off 7 Potential risk High High Low Medium Low High Low
- 10. 10© Oliver Wyman Distribution of model findings (1/2) Intentional simplifications and known limitations should receive strategic attention during validation, despite often being known prior to validation 56% 29% 12% Number of findings by category 72% 24% 4% Issues Simplifications OtherLimitations Impact of findings by category 3% <1%
- 11. 11© Oliver Wyman 79% 16% 4% Distribution of model findings (2/2) For companies with mature model validation functions, the majority of model findings are identified through model validation projects 76% 17% 5% 2% Audits Model change review Business unit review Model validation exercises 2% Number of findings by source Impact of findings by source
- 13. 13© Oliver Wyman Observations on model governance practices in the industry Our observations • Model governance standards are often burdensome to use • GAAP Targeted Improvements will be a catalyst for a wave of modernization initiatives • Vendor software packages offer superior features Implications • Propping up governance standards in an existing infrastructure is less effective • During larger model conversions or upgrades, the governance cycle has a chance to reboot and refresh • New software features can be leveraged to make model governance more efficient
- 14. 14© Oliver Wyman Model governance does not occur in a vacuum Mitigation of model risk should be base on: Model’s intended purpose1 Nature and complexity of the model2 Operating and control environment3 Model changes4 Balancing cost and risk reduction5 12 3 4 5 Source: Fourth Exposure Draft Proposed Actuarial Standard of Practice - Modeling
- 15. 15© Oliver Wyman What does an auditor look for when assessing model governance? What can be fixed relatively easily? HAVE A FLOWCHART ADDRESS GAPS HAVE A PLAN FOR FINDINGS CUT & JUSTIFY FLOWCHARTS HELP • Demonstrate how the inputs and processing components work together to meet the model’s intended purpose • Demonstrate where the controls are and how they work to mitigate model risk 1 GAPS DISTRACT • Some requirements may not apply • Spell out what does not and explain why not 2 PLANS ARE PROACTIVE • Completely shifts the messaging about findings • Enables audit to be a partner in reviewing and supporting plan 3 CUT TIME WASTERS • Governance activities may be added on over time without subtraction • Stop performing activities that don’t mitigate model risk (just because it appears on some generic requirement checklist) 4
- 16. 16© Oliver Wyman Common pitfalls in applying model governance CHAMPIONS WITH LIMITED INFLUENCE “IT’S NOT A MODEL” ONEROUS STANDARDS
- 17. 1717© Oliver Wyman Assigning responsibilities to the right groups and engaging IT Enterprise management IT services Business process 1 Business process 2 Business process 3 Business process 4 Enterprise controls Create a strong risk culture using: – Modeling standards – Governance policies – Code of conduct Application controls Verify completeness and accuracy of business processes with: – Authorizations/access – Approvals and sign-offs – Tolerance levels – Reconciliations – Change controls General controls Control shared services by performing: – Systems maintenance – Data management
- 18. 18© Oliver Wyman A sensible model governance architecture solves many but not all pitfalls TESTING ENVIRONMENT (MODEL STAGING) - UAT Model Model Data sources New business/ Product design Downstream processes Outputs ANALYTICS Quarterly inputs Data Data Data Assumptions and product features Data Data Data DEMOTION PROMOTION PROMOTION Adhoc analysis runs DEVELOPMENT PRODUCTION Intermediate inputs Intermediate inputs Ledger Downstream inputs
- 19. Model governance case study3
- 20. 20© Oliver Wyman Model governance case study Current state Identification Identify required model changes 1 Sandbox Testing Estimate impacts in sandbox model 2 Documentation Creation of model change documentation 4 Oversight Approval Review and sign-off by governance and risk oversight committees 5 Approved Implementation Implementation of model changes in the production environment 3 Run Production Run production model for financial reporting 6 Model Governance Hierarchy Risk Committee Model Owner Modeler Governance Committee
- 21. 21© Oliver Wyman 1 No defined review process The lack of an established independent review process increases risks of incorrect model change implementations Assign model steward and reviewers Assign a model steward to oversee the review process and individual reviewers to perform technical and peer reviews on model changes Potential pitfalls Proposed solution Model governance case study Pitfall 1: no defined review process
- 22. 22© Oliver Wyman Model governance case study Current state Identification1 Sandbox Testing2 Documentation 4 Oversight Approval 5 Approved Implementation 3 Run Production6 Model Governance Hierarchy Risk Committee Model Owner Modeler Governance Committee
- 23. 23© Oliver Wyman Identification1 Sandbox Testing2 Documentation 4 Oversight Approval 5 Approved Run Production6 Model governance case study Improved state – added review Model Governance Hierarchy Oversight Committees Model Owner Modeler Review Independent technical and peer review overseen by model steward Model Steward Implementation 3
- 24. 24© Oliver Wyman 1 No defined review process The lack of an established independent review process increases risks of incorrect model change implementations Assign model steward and reviewers Assign a model steward to oversee the review process and individual reviewers to perform technical and peer reviews on model changes Potential pitfalls Proposed solution Model governance case study Pitfall 2: implementation before approval 2 Changes are implemented before oversight approval Should a change be rejected, it will need to be reversed from the production model, introducing overhead costs and model risks Require approval for production model changes Proposed model changes should be tested and approved by governance committee before production implementation
- 25. 25© Oliver Wyman Model governance case study Current state Identification1 Sandbox Testing2 Documentation 4 Oversight Approval 5 Approved Implementation 3 Run Production6 Model Governance Hierarchy Risk Committee Model Owner Modeler Governance Committee
- 26. 26© Oliver Wyman Model governance case study Improved state – approval first Identification1 Sandbox Testing2 Documentation 5 Oversight Approval 3 Implementation 4 Rejected Run Production6 Model Governance Hierarchy Risk Committee Model Owner Modeler Governance Committee
- 27. 27© Oliver Wyman 1 No defined review process The lack of an established independent review process increases risks of incorrect model change implementations Assign model steward and reviewers Assign a model steward to oversee the review process and individual reviewers to perform technical and peer reviews on model changes 2 Changes are implemented before oversight approval Should a change be rejected, it will need to be reversed from the production model, introducing overhead costs and model risks Require approval for production model changes Proposed model changes should be tested and approved by governance committee before production implementation 3 Multiple oversight committees The existence of both governance and risk committees introduce additional overhead and may reduce efficiency of model change cycles Combine oversights and introduce model change governance criteria Combine oversight to a single committee and introduce separate documentation requirements and approval processes for changes based on materiality and complexity Potential pitfalls Proposed solution Model governance case study Pitfall 3: single governance oversight
- 28. 28© Oliver Wyman Model governance case study Current state Identification1 Sandbox Testing2 Documentation 4 Oversight Approval 5 Approved Implementation 3 Run Production6 Model Governance Hierarchy Risk Committee Model Owner Modeler Governance Committee
- 29. 29© Oliver Wyman Model governance case study Improved state – combined oversight Identification1 Sandbox Testing2 Documentation 4 Approved Oversight Approval Approval and sign-off by a single oversight committee 5 Implementation 3 Run Production6 Model Governance Hierarchy Oversight Committee Model Owner Modeler
- 30. 30© Oliver Wyman Potential pitfalls Proposed solution Model governance case study Summary of pitfalls and solutions 2 Changes are implemented before oversight approval Should a change be rejected, it will need to be reversed from the production model, introducing overhead costs and model risks Require approval for production model changes Proposed model changes should be tested and approved by governance committee before production implementation 1 No defined review process The lack of an established independent review committee increases risks of incorrect model change implementations Assign model steward Assign a model steward to perform technical and peer reviews on changes associated with each model 3 Multiple oversight committees The existence of both governance and risk committees introduce additional overhead and may reduce efficiency of model change cycles Combine oversights and introduce model change governance criteria Combine oversight to a single committee and introduce separate documentation requirements and approval processes for changes based on materiality and complexity
- 31. Wrap up and discussion questions4
- 32. 32© Oliver Wyman Discussion and Q&A What will model governance look like in an environment with increased automation? What makes certain model governance standards more effective than others? Any other trends in model governance that you see going forward? What are the top things to take away if you are going to participate in a model validation in the near future? 1 2 3 4