Building model trust
1 like348 views
Strong model risk management is important for insurers given their increasing reliance on complex models. Leading insurers develop MRM frameworks with policies for model governance, development and validation. Key aspects of model governance include defining models, maintaining an inventory, setting risk limits, and establishing roles and responsibilities. Models should be developed by experts in a controlled environment with thorough documentation. Independent validation ensures models are performing as intended, with priority given to riskier models. Validators check models against other independent models and outputs to identify any issues requiring remediation.
Building model trust
- 1. Technology Models are used broadly within insurance organiza- tions. For example, insur- ers use models to set premiums; develop appropriate reserve and capital levels as reported in finan- cial statements; evaluate potential divestitures and acquisitions; set business plans; and determine the types of assets to purchase. As a result, misuse of models or errors within models can have sig- nificant adverse consequences. Insurer models have become more intricate as computing power has increased, products have become more complex, asset/liability manage- ment more sophisticated and regu- latory requirements more onerous. Furthermore, for some insurers, par- ticularly reinsurers,the availability and quality of data might be a concern. For innovative or small insurers, there may not be credible expe- rience supporting the model- ing assumptions. Also, new types of models are continuously being developed and used (such as pre- dictive models in life insurance). These are just some of the factors that contribute to increased model risk within the insurance industry. While most insurers have invested in integrated risk management for a long time, model risk management has recently become an area of emphasis for some insurers. Leading companies have devel- oped MRM frameworks to mitigate model risk and to become more comfortable with their models. Common elements of MRM frame- works include policies related to model governance, development and implementation,and validation. Model Governance Strong model governance is criti- cal to effective MRM.The boards of directors and senior management at best-in-class companies have recog- nized this by sponsoring, support- ing or leading the establishment of MRM frameworks at their compa- nies to help foster an MRM culture. An MRM framework is an impor- tant component of a company’s overall risk management approach. As such, the steps involved in devel- oping an MRM framework are simi- lar to the steps used to develop an ERM framework. For example, a critical element of any risk management approach is to identify a company’s specific risks. A company can determine model risk exposures by defining what a model is, creating and maintaining an inventory of models, and categoriz- ing the models into low, medium and high risk with due consideration for materiality. Also, a company would normally set risk limits to ensure that the degree of model risk does not exceed a specified risk tolerance. Other important elements of governance related to MRM include policies or best practices related to model development, testing, docu- mentation, approvals, version and change control, access rights and validation. Roles and responsibilities are another important governance consideration. Ideally, model devel- opment and maintenance duties would be segregated from model user duties. The internal audit or risk management departments are often involved in measuring the effectiveness of MRM governance. Some companies have estab- lished MRM officers to lead the model risk management function. Development and Implementation MRM begins with model devel- opment. Ideally, models would be created by experienced developers and subject matter experts using high-quality data, robust methodol- ogy and assumptions that are sup- ported by credible experience in a controlled modeling environment. Models in a controlled environ- ment allow for adequate testing by Dom Lebel and Sebastien Cimon Gagnon Strong model risk management can prevent the damaging impact that model misuse or errors have on insurers. Building Model Trust Contributors: TowersWatson’s Dom Lebel is a director responsible for financial modeling and reporting for the Americas Life practice;and Sebastien Cimon Gagnon is a senior actuarial analyst.They can be reached at dominique.lebel@towerswatson.com and sebastien.cimon.gagnon@ towerswatson.com Lebel Gagnon Key Points ▼ The Issue: Insurance companies are using models that are more complex than ever. ▼ The Problem: Even slight modeling missteps can have an enormous impact on an insurance company’s financials, risk management or decision-making. ▼ The Solution: Model risk manage- ment provides a structured, objective system for making sure they’re reliable. Copyright © 2014 by A.M. Best Company, Inc. All Rights Reserved. Reprinted with Permission. www.ambest.com BEST’S REVIEW www.bestreview.com September 2014 BEST’S REVIEW • SEPTEMBER 2014 • REPRINT 1
- 2. Technology and approvals before implementa- tion. Model documentation would be sufficiently detailed to describe the purpose of the model and how it will be used; data inputs to the model; and model methodology and assumptions, including their ratio- nale. Documentation also would detail the model’s compliance with regulations or other standards, the testing performed and known mod- el limitations. Updates and changes to models would be governed by a version- and change-control process. While some companies have developed standardized templates to facilitate consistent model doc- umentation, resource constraints, company culture or inadequate model governance can make it a struggle for many. Cases with inad- equate documentation of models (in particular, no documentation of model limitations) have resulted in models being used for purposes they were not designed for. We recommend structuring the documentation so it is easily main- tained when the model is updated and revised. Model documentation ensures the model was adequately devel- oped; model owners are account- able; and model users understand how the model operates, how model results should be used and any limitations. Model Validation The intended purpose of model validation is regular verification that models are performing as expected. While essential, this process is time- consuming and resource-intensive.As such, priority generally is assigned to newer models,models with the great- est degree of model risk and models that have not recently been reviewed. To avoid conflicts of interest, the validator should be objective; knowledgeable about the product being modeled, modeling standards of practice, industry practices and regulatory requirements; and not have an interest in the outcome of the validation. For example, model developers or users should not be validators.As a result, model valida- tions are often performed by con- sultants. Accountability will also be encouraged if there is a sole valida- tor per model. Some companies have devel- oped model validation templates for validators to use. The translation of relatively high-level validation requirements to detailed and prag- matic validation activities can be challenging. Regardless of whether a valida- tion template is available,model vali- dators generally develop a validation test plan.All model components are typically validated, including inputs, calculations and output. An MRM governance review is also typically within the scope of a model validation project. It focuses on the application of MRM gover- nance policy; modeling oversight by senior management and the board; establishment and adherence to model risk limits; maintenance of the model inventory, roles and respon- sibilities; and various other model controls such as access rights,change processes and approval procedures. The validator typically starts by reviewing existing model documen- tation, including support for model- ing assumptions and testing that has been performed to evaluate concep- tual soundness. It is also important for model vali- dators to understand the purpose of the model and how it is currently being used, because confirming that the model is being used as intended is an important validation element. This is even more critical when no support for assumptions exists due to lack of company experience or indus- try data. Next, the validator typically checks model output against another model that has been independently developed to confirm that the mod- el has been properly implemented. Since full reconciliations of model output between two independent models can be time-consuming,toler- ances often are used to minimize the amount of work. Validators will often perform additional checks such as compar- ing model output to historical, actu- al audited results and testing output sensitivity to assumption changes. We have found it useful for valida- tors to maintain a review log that categorizes types of issues (such as approximation,error and misuse). The review log can also contain an estimate of the impact of each issue, which can be assessed by the validator with support from model developers or users. Governance The Model Risk Management Cycle Model validation Model development and implementation Use Key Model Validation Principles 1. Model design and build need to be consistent with the model’s intended purpose. 2. Ensure that model validation is an independent process. 3. Establish an owner of model validation. 4. Ensure appropriateness of established model governance. 5. Make model validation efforts proportional to evidenced areas of materiality and complexity. 6. Validate the model components. 7. Address limitations of model validation. 8. Document the model validation. Source: The North American CRO Council 2 BEST’S REVIEW • SEPTEMBER 2014 • REPRINT Copyright © 2014 by A.M. Best Company, Inc. All Rights Reserved. Reprinted with Permission. www.ambest.com
- 3. Thereafter, the validator prepares a model validation report that clearly details the testing performed;the find- ings from the validation; recommen- dations for improvement;a prioritized ranking of next steps; and limitations of the validation. An executive summary is often prepared for senior management to summarize key findings. Models that contain issues can be updated,replaced or restricted to cer- tain purposes, depending on the company’s risk limits and material- ity thresholds. Likewise, any issues related to gov- ernance of the model could lead to changes in model governance policies. When models are changed, this marks the beginning of a new mod- eling cycle of development and implementation (which includes documentation, testing and approv- als),use and validation. Why MRM Matters Models are now intrinsic to the operations of insurance compa- nies. Their misuse or errors can have a significant, harmful impact on an insurer. And over time, mod- els have become increasingly com- plex, making it even more impor- tant that insurers pay attention to model risk. Leading insurers are developing MRM frameworks that include mod- el governance, development and implementation, and validation to mitigate model risk in response to external forces such as regulators, or proactively as companies become more aware of the potential impact of model risk. Actual or perceived conflicts of interest can diminish an insurer’s efforts, so it is important for objectivi- ty to guide MRM activities.When done right, MRM can provide confidence to internal and external stakeholders that a company’s models are sound and that their results are reliable. BR Inside and Outside Influences Model risk management typically is driven by some or all of these factors: • Increased frequency of financial restatements due to model errors. • Ineffectiveness of some models connected with the recent financial crisis. • Sarbanes-Oxley requirements assigning management responsibility for financial statements. • Own Risk and Solvency Assessment requirements developed by the National Association of Insurance Commissioners for insurers to self-assess their risk management adequacy and prospective solvency under normal and severe stress scenarios. • Board of Governors of the Federal Reserve System SR 11-7 Supervisory Guidance on Model Risk Management. • Solvency II and its Internal Model Application Process. • The North American CRO Council’s Model Validation Principles Applied to Risk and Capital Mod- els in the Insurance Industry. • Greater focus on models by internal and external auditors and regulators. • Management’s desire to understand model limitations and weaknesses. Source: Towers Watson Copyright © 2014 by A.M. Best Company, Inc. All Rights Reserved. Reprinted with Permission. www.ambest.com BEST’S REVIEW • SEPTEMBER 2014 • REPRINT 3