Model Risk Report : simplebooklet.com

QuantUniversity Quantifying Model Risk Issues and approaches to measure and assess model risk when building quant models ematical theories techniques and assumptions to process input data into quantitative estimates They note that model risk occurs if the models have fundamental errors and produce inaccurate outputs or if a model is used incorrectly or inappropriately The note further elaborates on what is expected of supervisors and organizations in managing model risk Recently we have seen a significant increase in organizations taking up model risk management initiatives Let s discuss some drivers for this emphasis on model risk W ith model failures leading to some high profile financial accidents in the past few years there has been a renewed emphasis to systematically address model risk in the last few years Regulatory agencies have issued guidance documents and various organizations including QuantUniversity are offering model review and validation services to assist organizations to comprehensively manage model risk for their quantitative models With the financial industry relying heavily on quantitative models and with serious repercussions for model failures it has become essential that organizations prioritize model risk management Companies are recognizing the need to develop robust processes to address various aspects of risk evolving from the use and deployment of their quantitative models Though a lot has been written about model risk management there is little discussion on the practical tools quants could use to measure and assess model risk as part of the quant development process In many of the consulting projects we have worked on model risk management is typically an afterthought usually delegated to the risk organization after quants have completed their development work This handoff increases the possibility of model failures and the consequences could be too acute to remedy At QuantUniversity we believe that the measurement and assessment of model risk should be an integral part of the quant development process In this article we outline some of the tools quants could use to integrate the measurement and assessment of model risk as part of their development process We start out 56 Drivers to address model risk Increase in the sophistication and reliance on quantitative models by orienting quants on model risk and discussing drivers that have brought model risk into the limelight We then discuss challenges quants have in integrating model risk into the quant development process and introduce practical tools quants can use to quantify model risk and build control measures to manage and prioritize these risks Model risk Quantitative models are in the critical path of decision making for various organizations today In the areas of trading risk management credit rating and asset management quantitative models are at the heart of many processes and any malfunction or failure in models can lead to reputation risk huge opportunity costs and or financial losses As more and more organizations rely on quantitative models for decision making it is important to ensure that the models perform as per design The Federal Reserve and the Office of the Comptroller of the Currency OCC issued a Supervisory Guidance on Model Risk Management 1 for banks to assess model risk In this note they define a model as A quantitative method system or approach that applies statistical economic financial or math Advances in computing technology faster networks and connectivity and the maturing of quantitative technologies have led to a significant increase in using quantitative models in the financial industry Advances in financial engineering have led to newer and more sophisticated financial products that cannot be constructed and used without the help of quantitative models and analytics As the reliance on these models has increased risk departments have started to recognize the importance of managing risk due to the use of quantitative models in organizations Robust software development practices and security protocols are being established and control mechanisms are being put in place to manage model risk Regulation With the financial crisis of 2008 regulators are sensing an urgent need to address model risk management and there have been many key initiatives toward facilitating this The Dodd Frank Act Basel and the Solvency frameworks explicitly address model risk and validation issues that have required banks and insurance companies to institute processes to manage model risk and develop comprehensive model risk management frameworks As noted earlier the Federal Reserve and the OCC saw the need to issue Supervisory Guidance wilmott magazine

Page 4

on Model Risk Management that discusses model development implementation use validation governance etc in significant detail With the stakes so high and with bitter lessons learnt during the financial crisis there is increasing regulatory oversight to ensure the stability of the financial system and this oversight is expected to increase in the future However it should be noted that despite regulatory efforts and guidance organizations are still facing practical challenges in comprehensively establishing a model risk management framework For example the FSA s Internal Model Approval Process Thematic Review 2 discusses some of the issues and challenges faced by organizations in adopting the Solvency II framework requirements Financial accidents In the recent past we have seen many instances where model risk has led to devastating outcomes for organizations For instance S P disclosed errors in their rating models 3 and faced severe criticism for their ratings approach during the financial crisis Knight Capital suffered a 457 5 million loss due to a software error 4 and Goldman Sachs issued erroneous options orders due to a software bug that cost the firm millions of dollars 5 Note that these are reputable players in the financial industry but still had to face enormous consequences both financially and in terms of their reputation With high frequency trading constituting more than 70 percent of equity trades in the USA it is practically impossible for humans to intervene and plug damage when millions of trades are executed in a matter of seconds The magnitude and impact of erroneous models have alerted the financial industry to be serious about addressing and instituting processes to manage model risk wilmott magazine Challenges to integrating model risk assessment into the quant development process Recently we have seen several organizations embark on developing model risk management programs within their organizations For many the drivers are purely regulatory but some organizations have highlighted the other drivers we listed before as reasons for developing model risk management programs In discussions with quant groups we uncovered multiple challenges that quants deal with when assessing and addressing model risk in their quant development process Organizational structure Many organizations consider model risk management as an initiative driven by the risk group in their organization But the risk groups typically don t have the granular details of the model design and implementation and typically have to involve the quant groups to help articulate the various risk aspects and to devise methodologies to address model risk issues In many cases model risk management is an afterthought and risk departments try to build processes to meet regulatory requirements which don t address the spirit of model risk management In addition organizations have multiple groups working on different aspects of the quant development process A quant research group may be in charge of vetting models and their application A quant development group may be in charge of developing models The IT group may be involved in deploying these models and a risk group may be expected to have oversight on all these departments Identifying and delegating responsibility for model risk programs requires significant coordination between departments This is a fairly new process and best practices are still evolving on effective ways of identifying design ing and managing organization wide model risk management programs Practical challenges How to do it There is a lot of research and literature available on what constitutes a robust model risk management framework Regulatory guidelines consulting companies and industry organizations like the CFA Institute and PRMIA have made available recommendations that help define the key tenets of a model risk management framework One of the primary challenges that quants deal with is practical knowledge on how to incorporate model risk management requirements into their model development process This includes identifying potential risks in their models quantifying them building controls to address these risks and prioritizing model enhancements to address risks Since model risk management programs are an organization wide effort quants are expected to design and implement artifacts to address the identified model risk aspects For example model validation is a key test to be carried out by insurers as part of the Solvency II framework 6 and many organizations have had challenges in implementing these requirements The recommendations may not be easily translatable into actions that quants can easily integrate into their quant development process Typically the recommendations and guidelines are qualitative and there are few guidelines and tools for practical implementation In addition prioritizing the identified model risk addressing these risks through model changes assessing the impact of these model changes to address model risk and other consequences such as speed performance etc of a model is complicated without proper guidelines and metrics When there are multiple issues to be addressed in a model quants need to figure out ways to prioritize issues and assess the impact of interactions between issues and any changes they implement in the model Since the field of model risk management is relatively new organizations rely on experts and consultants to help them formulate and execute a model risk management program Model review and validation services are typically sought by companies to help quant groups implement a model risk management program Companies including QuantUniversity typically work with quant groups to help design and operationalize model risk management practices Competitive advantage Many organizations especially trading firms use quantitative models as a key competitive advantage in leveraging opportunities in the market In domains such as algorithmic trading automated models rely on fast algorithms to quickly take trading decisions capturing opportunities in sub seconds while competing with other market participants Speed and efficiency are important and companies spend significant time in optimizing algorithms before they are deployed in production In certain instances multiple development and test cycles are involved and algorithms are recompiled into lower level languages or hardware specific languages to take advantage of the execution environment Models which run on GPU and FPGA architectures are examples of this These organizations are quickly realizing that model risk needs to be managed throughout the development testing and execution process to sustain their competitive advantage 57

Page 5

QuantUniversity Tools for quants to integrate model risk assessment into the quant development process In this section we discuss an approach we recommend quant groups use to quantify and manage model risk It builds upon the aspect impact analysis framework advocated in the ISO 14001 standard and also popular in IT enterprise risk management and computer security risk assessment frameworks 7 We use this methodology to help quants identify issues in models quantify the risks and associate model risk controls to help mitigate or manage model risk score The difference between the old risk scores and the new risk scores indicates the risk exposures that are still to be addressed which could be used to reinforce the model risk controls Figure 2 shows a sample template to list the aspects impacts and model risk controls and to calculate risk scores Column D lists the various aspects pertaining to a model Column E lists the associated impacts column F is used to rate the impact scores column G to note the probability of occurrence and column H to derive the risk scores Column I is used to list the model risk controls implemented to address the potential impacts and a new risk score is computed in column L Finally the residual risk exposures are listed in column M The risk scores help in implementing or validating if the model risk controls are sufficient to handle the risks posed by various aspects For example in Figure 3 we illustrate the possibilities of risk scores for an impact score ranging from 1 5 and a likelihood rating of 1 5 A quant group may categorize four kinds of aspects to determine priority and actions for model risk control measures For example High Impact High likelihood of occurrence Needs adequate model risk control measures to mitigate risk High Impact Low likelihood of occurrence Addressed through model risk control meas Model risk assessment framework When we work with organizations to help structure a model risk management framework we recommend organizations form internal model review teams to work with quants in their model risk assessment efforts The review team typically includes the quant research team development team subject matter experts or architects who are familiar with the modeling methodology but not part of the implementation group other stakeholders including IT and risk management The goal for this team is to identify and quantify the key aspects that would lead to model risk the potential impacts these aspects could have and the model risk controls implemented or to be designed to handle these risks Figure 1 describes the key terminology used in this approach The review team reviews every quant model and identifies aspects Aspects are model issues that lead to potential model risks Figure 1 Key terminology in the model risk assessment Then the team lists the potential impacts framework the aspects could cause Impacts are undesired consequences that need to be eliminated or mitigated Scoring guidelines Once the aspects and impacts are identified the next step is to rate the impacts and estimate the probability of occurrence of these impacts The goal is to derive risk scores for each aspect which factor the impact scores and the likelihood of occurrence of these impacts as defined in Figure 1 Then the model risk controls implemented to address these aspects are considered and a new impact score and likelihood of occurrence is rated to develop a new risk 58 ures and contingency plans L ow Impact High likelihood of occurrence Lower priority model risk control measures L ow Impact Low likelihood of occurrence Least priority model risk control measures Based on the scores the quant group may classify aspects and designate red yellow and green colors to indicate categories of high risk moderate risk and low risk respectively The group may then decide that models with red category aspects cannot be deployed till there are sufficient model controls in place This method also helps prioritize the aspects and controls that need to be implemented to address model risk In addition this approach helps invoke discussions within the quant group and in the model internal review team to ensure there is agreement on the risk perception and risk appetite and to ensure that risk controls implemented are vetted thoroughly Owners can be designated to address these risks and timelines noted to ensure that the risk exposures are mitigated in a timely manner A key advantage of using this framework is that the structured and team based approach enhances transparency in the quant group and other stakeholders while alleviating the risk of unforeseen events This structured approach also helps identify training opportunities and evolve best practices for the development process in the quant group to continually improve the model development process Note that this process isn t a one time activity and periodic reviews need to be carried out The triggers could be new model issues changes to existing models or model enhancements that introduce material changes to quant models We have designed tools and dashboards to track changes and provide a current view of the existing state of model risk in the group and update the state when changes are introduced This facilitates various stakeholders in the organizations to know and review progress and plan to address gaps It also helps in recordkeeping and communicating the current state of model risk management to external parties magazine

Page 6

QuantUniversity Figure 2 Sample risk grading considering impact and likelihood of occurrence including consultants and regulators Though this framework doesn t cover all aspects of model risk management it provides quant groups a practical and easy way to assess model risk and helps build the foundation to integrate other aspects of model risk management into the quant development process Summary Model risk management is becoming a key function for quantitatively driven financial organizations In this article we briefly introduced model risk and discussed the drivers and reasons to address model risk We also discussed some of the challenges quant groups face when integrating model risk assessment into the quant development process We then discussed the model risk assessment tool to help identify issues in models quantify the risks and associate model risk controls to help mitigate or manage these risks We illustrated through templates the tools that quants can use to have a structured approach to model risk assessment and discussed some of the advantages of this approach By integrating model risk assessment into the quant development process quants can build robust models and ensure that model risk is effectively managed within their organizations Figure 3 Scoring guidelines template Impact 5 4 3 2 1 Red Yellow Green wilmott magazine 5 4 3 2 1 1 High Risk Moderate Risk Low Risk Risk Scores 10 15 20 8 12 16 6 9 12 4 6 8 2 3 4 2 3 4 Likelihood of occurrence References 1 OCC 2011 12 SR 11 7 Supervisory Guidance on Model Risk Management 2 http www fsa gov uk pubs international imap_final pdf 3 http www reuters com article 2008 06 13 sp ratings credit idUSL1363473320080613 4 http www bloomberg com news 2012 10 17 knight capital reports net loss as software error takes toll 1 html 5 http online wsj com news articles SB10001424127887324 747104579024964124614096 6 http www risk net insurance risk news 2171368 slow progress solvency ii internal model validation threatens approval fsa warns 7 http csrc nist gov publications nistpubs 800 37 rev1 sp800 37 rev1 final pdf About the Author 25 20 15 10 5 5 Sri Krishnamurthy CFA CAP is the founder of QuantUniversity com a data and quantitative analysis company He has significant experience in designing quantitative finance applications for some of the world s largest asset management and financial companies He teaches quantitative methods and analytics to MBA students at Babson College and is the author of the forthcoming book published by Wiley titled Financial Application Development A Case Study Approach QuantUniversity offers quantitative modeling and consulting services to financial institutions QuantUniversity recently launched a new offering called Model Risk Review and Validation Service and offers training and custom consulting packages for model risk management For a copy of the sample templates quants can use for model risk management contact Sri at sri quantuniversity com 59

Page 7

QuantUniversity The Decalogue Ten best practices for an effective model risk management program models and the methodologies involved review of the model development life cycle and review of existing company policies and regulatory requirements that need to be followed The three minimum things a model risk management framework should incorporate include Model governance structure Addresses regulatory requirements roles responsibilities oversight control and escalation procedures Model life cycle management Addresses the processes involved in the design development testing deployment and use of models Also addresses testing and documentation plans and change management Model review and validation process Addresses internal and external model review validation and ongoing monitoring of models both qualitative and quantitative Defining a framework ensures appropriate bucketing of tasks and a structured approach to developing a model risk management program M odel risk and the importance of model risk management have gotten significant attention in the last few years As financial companies increase their reliance on quants and quantitative models for decision making they are increasingly exposed to model risk and are looking for ways to mitigate it The financial crisis of 2008 and various high profile financial accidents due to model failures 1 3 have brought model risk management to the forefront as an important topic to be addressed Many regulatory efforts Solvency II Basel III Dodd Frank etc have been initiated obligating banks and financial institutions to incorporate formal model risk management programs to address model risk Regulatory agencies have issued guidance letters and supervisory insights 4 5 to assist companies in developing model risk management programs In the United States as the Dodd Frank act is implemented newer guidance letters for example 6 have been issued that emphasize model risk management Despite these efforts in practice financial companies continue to struggle in formulating and developing a model risk management program A lot of companies acknowledge and understand the model risk management guidelines in spirit but have practical challenges in implementing these guidance letters In our prior article on model risk 6 we discussed many drivers to address model risk and challenges in integrating model risk into the quant development process In this article we will discuss ten best practices for the implementation of an effective model risk management program These best practices have 58 evolved from discussions with industry experts and consulting projects we have worked with in recent years to create robust risk management programs These best practices are meant to provide practical tips for companies embarking on a formal model risk management program or enhancing their model risk methodologies to address the new realities Adopt a framework driven approach for model risk management A model risk management program must obtain company wide support and buy in from all stakeholders engaged in model development use validation and compliance for it to be successful Adopting a framework driven approach ensures that all elements of the model risk management program are defined clearly before embarking on a company wide initiative This includes identification of the stakeholders identification of the Customize a model risk management program An additional advantage of a framework driven approach to model risk management is that the program can be customized to the institution s needs Every organization is different The size the structure and the activities that an organization engages in generate different levels of risk exposure In addition every company has its own company culture and core values and adopting any new company wide initiatives is challenging Adopting a model risk program could change the way models are developed governed and used in an organization Quant departments may feel that a model risk program s need for structure transparency and oversight imposes constraints and overheads that didn t exist before To ensure the success of a model risk management initiative the program should be customized considering the structural aspects along with the nuances specific to the organization This doesn t mean diluting the requirements of the wilmott magazine

Page 8

model risk management program Customization requires effective engagement of all stakeholders and an internal agreement to the spirit of model risk management Recognition of the need to incorporate changes to existing model development methodologies and buy in from all groups is needed to reduce potential friction between departments We recommend that companies take a holistic approach to ensure that the model risk management principles and regulatory requirements see the Supervisory Guidance on Model Risk Management 4 are effectively incorporated in a model risk management program that is customized to the organization s needs and structure Figure 1 Model life cycle management Clearly define roles and responsibilities wilmott magazine ment program Committees need to be established to facilitate decisions Organizations may also have to make changes in the organizational structure to create roles for model risk and validation related responsibilities For organizations embarking on new model risk management initiatives internal model risk expertise may not be available and external candidates who have experience in model risk may have to be hired to facilitate model risk programs In those situations external model validators could fill the gap in facilitating model review and validation They could also provide the best practices needed to structure and enable a model risk management program see 8 for our example offering for the long run Integrate model risk management effectively into the model life cycle The life cycle of a model see Figure 1 begins at its conception requirements drawn from a methodology prototyping testing documentation review and validation production ongoing monitoring and model enhancements or retirement if it has met its purpose The goal of model risk management is not to have an additional step in the model life cycle as a gating process for model approval approve not approved before releasing a model to production As model risk drivers are permeated throughout the model life cycle for effective model risk management the model life cycle must integrate model risk processes needed to check the validity and effectiveness of the model into the model life cycle Many a time model risk management is bundled into the model review and validation step in the model life cycle and models are submitted to the model validation team for review and validation after the model is completely built As many model risks can be effectively identified and mitigated through addressing of the issue or by putting controls in place during the model development process the model development and use guidelines must integrate measures to address model risk in every step of the model life cycle Don t reinvent the wheel Some of the best practices in model risk management have been in vogue in other areas for decades In our last article 7 we discussed an approach to quantify model risk A similar approach for quantifying environmental risk was used when I was an engineering trainee in a company that manufactured hydraulic excavators in the 1990s As a A framework driven approach to model risk management also facilitates clear articulation of the roles and responsibilities of all stakeholders Note that a company wide model risk management program should engage all stakeholders from those who are responsible for designing developing and using the models to those who are in a supervisory capacity and are accountable for the models The OCC supervisory guidance 4 discusses the notion of an effective challenge of models which it defines as the critical analysis by objective informed parties who can identify model limitations and assumptions and produce appropriate changes In order to facilitate an objective and independent assessment of model risk organizations must set up structures to segregate groups engaged in the design and development of models from groups who are responsible to pose an effective challenge through model review and validation activities As per the Supervisory Guidance on Model Risk Management 4 key considerations for effective challenge include incentives competence and influence To create a structure within the organization that would make an effective challenge feasible senior management must be engaged and must provide the budget and resources to ensure sufficient delineation of responsibilities and the operational and governance structure needed to implement a model risk management program At times an organization may not have the expertise in model risk management and resources to take on roles to facilitate an effective model risk manage 59

Page 9

QuantUniversity part of ISO 14001 certification we were quantifying and putting controls in place to mitigate the risk of phosphate sludge entering storm water runoffs The point is model risk management practices are not new and we should leverage and augment the best practices from other areas rather than reinventing the wheel For example many IT groups would already have the infrastructure and established processes for software development These would typically incorporate software development best or Python and then translated to C C for performance Models may run standalone on desktops or may have been deployed onto servers Considering all these possibilities it is impossible to have one standard for all models We advocate categorizing the models in the model inventory into classes and have a class based model policy for model risk management In the consulting projects we have worked on we typically create an inventory of models and segment and The point is model risk management practices are not new and we should leverage and augment the best practices from other areas rather than reinventing the wheel practices for security management change management configuration management testing logging archiving etc Rather than creating new processes and standards for quantitative models existing methodologies can be integrated into the model life cycle effectively creating robust practices for development and to control risk We advocate engaging IT departments in your companies as an important stakeholder to leverage their experience and best practices in developing a model risk program All models weren t born equal As per the Supervisory Guidance on Model Risk Management 4 a model is defined as follows Model refers to a quantitative method system or approach that applies statistical economic financial or mathematical theories techniques and assumptions to process input data into quantitative estimates An organization may have models ranging from simple spreadsheets to models that are complex and require simulations and optimization routines for implementation Some models may be homegrown internally developed and some may be sourced through consultants and vendors The model development methodology may require models to be prototyped first in fourth generation languages like MATLAB R 60 score models based on criteria such as complexity dependencies impact maturity etc We then categorize them into classes Class 1 2 3 etc and formulate policies or model risk procedures for each class of models The criteria for classification depend on the kind of models a company uses for decision making Classification has to be done after a thorough understanding of the context and model use cases A checklist is Your friend Atul Gawande s best seller The Checklist Manifesto 9 discusses and advocates the use of the simple concept of checklists that enable doctors handling complex cases to develop treatment plans effectively Doctors work in extremely stressful situations and a simple mistake could cost human life Incorporating checklists increases efficiency in the process and allows doctors to focus on important decisions while relying on checklists for protocols to be followed We advocate using checklists throughout the model risk management program Checklists ensure that the process is documented and that quants have a repeatable and reliable process to follow when developing models They provide traceability and reliability to processes and help meet documentation and archiving requirements in model risk management programs In addition checklists ensure that risk mitigation controls can be triggered systematically when models aren t performing as per expectations Checklists also help in finding gaps in the process and to improve the robustness in processes Having checklists to follow when a model is unusable either due to model specific issues such as model crashes software bugs or when there are external adverse unpredictable events market crashes geopolitical events etc is extremely useful It can provide departments with a structured way of implementing policies during stressful and unpredictable situations to ensure business continuity We advocate building checklists and templates in each step of the model life cycle and review and validation steps to enhance the robustness of the model risk program An inventory of checklists and templates ensures that the process is standardized and that a systematic approach to model risk management is followed helping model review and validation efforts Monitor the health of the models and the program Ongoing monitoring is a critical piece of a model risk management program As the old management adage goes You can t manage what you don t measure Model risk management is an ongoing process and is not limited to periodic model checks or exception handling The models in use should continually be monitored and as time passes the models may have to be recalibrated to accommodate new market or business conditions The assumptions incorporated in a model may no longer be valid and models may not perform as designed In addition errors may be uncovered and new business requirements may be needed We advocate defining a thorough process for measuring and monitoring the health of the models Systems and processes to document observed changes in model behavior and risk characteristics must be defined and tested as a part of the model risk management program Periodic reviews at least annually of the health of the models must be conducted to ensure that the model characteristics are acceptable If the model performance is deemed unacceptable or if the model risk has changed suitable measures must be incorporated to ensure that the risks are mitigated Appropriate escalation procedures must be documented and roles identified to ensure that the risks are communicated through proper channels magazine

Page 10

QuantUniversity Leverage your domain knowledge on the models Ultimately you know more about the models you use than anyone else outside your company A model risk management program is successful when an organization is involved throughout the life cycle of model development and understands thoroughly how to put a model risk program in place It is a common practice to engage consultants and vendors to help build your models Although this may be inevitable for example when you are working with a new vendor or don t have the expertise to build your own models we advocate training and ramping up of internal resources who can be fully engaged in the model life cycle You know your models best Vendors and consultants may not have the domain experience especially if they sell platform products which support multiple domains I once worked with a consultant who wanted help with a VaR model Hearing his use case when I asked whether it was the Vector autoregressive model or the Value at Risk model he wasn t sure which one it was I worked with another engineer who developed a function called getGreeks This function took one input and returned one of the Greeks Unfortunately the input variable was encapsulated in another object and until you manually debugged the code you would have no idea which Greek was returned as an output When vendors third party consultants provide minimum documentation and don t have the required domain knowledge it is the responsibility of the model development group to set expectations and standards for the models being built for them Even if you use models developed by vendors you are ultimately accountable and need to put together a thorough plan to ensure that the vendor models are tested prior to deploying those models into production We advocate noting dependencies for each model in your inventory and ensuring that you have a thorough plan in place when leveraging vendor or third party developed software Own the model risk management program Initiating a model risk management program is not about satisfying regulators or external validators or even just the model validation group in your company A systematic approach to model risk man wilmott magazine agement not only mitigates risk but also provides a source of competitive advantage Owning the model risk management program and integrating it as a part of the model life cycle ensures that the adverse effects due to model failures can be mitigated especially during unpredictable and stressful situations It also creates a transparent and resilient model development framework and an awareness within and outside of the model development group of the state of model development as well as helping to plan strategic changes to the model development and validation processes In addition an enterprise wide model risk management program with full support from senior management demonstrates commitment and seriousness toward creating an effective model risk methodology A sense of ownership among various stakeholders aligns stakeholders to not only adhere to the requirements of the program but also to proactively engage in continuously improving the program to make it more robust Conclusions Model risk management has recently gained significant attention in financial institutions Initiating and adopting a model risk management program can be daunting considering the lack of established best practices and limited guidance available to institutions With new regulations mandating strict policies to address model risk institutions are required to initiate formal model risk management programs As more and more companies are embarking on model risk programs it is essential that they consider some of the best practices and successes that other organizations have had before embarking on a company wide model risk management initiative In this article we have highlighted ten best practices that institutions should consider when designing and implementing a formal model risk management program These best practices have evolved from discussions with industry experts and practical experience gained through working with multiple customers By incorporating these best practices companies can develop a robust model risk management program that provides a strong foundation for model life cycle management while effectively addressing the challenge of model risk References 1 http www reuters com article 2008 06 13 sp ratings credit idUSL1363473320080613 2 http www bloomberg com news 2012 10 17 knight capital reports net loss as software error takes toll 1 html 3 http online wsj com news articles SB100014241278873247 47104579024964124614096 4 OCC 2011 12 SR 11 7 Supervisory Guidance on Model Risk Management 5 http www fdic gov regulations examinations supervisory insights siwin05 article01_model_governance html 6 http www federalreserve gov bankinforeg srletters sr1403 htm 7 Krishnamurthy S 2014 Quantifying model risk Wilmott 2014 69 56 59 8 www quantuniversity com modelrisk 9 Gawande A 2010 The checklist manifesto How to get things right Metropolitan Books About the Author Sri Krishnamurthy CFA CAP is the founder of QuantUniversity com a data and quantitative analysis company Sri has significant experience in designing quantitative finance applications for some of the world s largest asset management and financial companies He teaches quantitative methods and analytics for MBA students at Babson College and is the author of the forthcoming book published by Wiley titled Financial Application Development A case study approach QuantUniversity offers quantitative modeling and consulting services to financial institutions QuantUniversity recently launched a new offering called Model Risk Review and Validation Service and offers training and custom consulting packages for model risk management For a copy of sample templates that quants can use for model risk management contact Sri at sri quantuniversity com 61

Page 11

QuantUniversity Not by Faith The role of model verification in model risk management Introduction The financial crisis of 2008 and the regulatory initiatives since then have put the spotlight on financial models and the need to manage model risk in financial institutions More and more financial institutions are formally incorporating model risk management programs and are changing model development testing and deployment processes to adhere to regulatory guidelines and best practices As the field is relatively new and best practices are still evolving organizations have had to try out various initiatives to customize model risk management programs that balance regulatory requirements and organizational objectives As model complexity increases model risk management becomes difficult as only a few groups in the organization know what a model actually does Translating model designs to model implementation brings in additional challenges First model development teams must ensure that they understand the model designs and can implement the designs accurately For complex and proprietary financial products quants find it challenging to communicate the design intent and appropriate use of quant models Second ensuring that the model in question is implemented properly as per intended design and validating that the outputs are representative of the characteristics they are modeled after is complicated For computer simulation models these activities are addressed in model verification and model validation 1 activities Financial organizations are realizing the need to integrate rigorous model verification and model validation processes prior to deployment to understand and mitigate model risk Although a lot of literature and practical guidelines are available for model validation very little is available for model verification for financial models 56 In our previous article 2 we discussed best practices for the implementation of an effective model risk management program In the this article we delve deeper into the role of model verification in model risk management We start out by discussing the differences between model verification and model validation We then discuss why formal model verification is necessary and make a case for integrating formal model verification as a part of model risk management We then discuss four key elements that need to be considered as a part of a formal model verification process We hope that through this article we can highlight the importance of model verification in building a robust model risk management practice Model verification and model validation Although the concepts of model verification and validation are relatively new in finance it is a well established science and is widely adopted in fields such as design engineering and manufacturing Whether it is designing structures that support skyscrapers or crash tests for cars computer models are built to simulate actual conditions and to test how the model would behave when exposed to various input parameters and stresses During this process verification and validation is done to determine compliance to design and to determine if the model behaves as a real world process would One of the best definitions of verification and validation is found in a Department of Defense note 0 Here verification is defined as The process of determining that a model or simulation implementation and its associated data accurately represent the developer s conceptual description and specifications Validation is defined as The process of determining the degree to which a model or simulation and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model Note that the primary goal of verification is to ensure correctness and to determine if the implementation accurately reflects the design specification The goal of validation is to determine how closely the model reflects the behavior of the process product it represents In financial model risk management many times verification and validation are clubbed under the umbrella of model validation Morini 4 was one of the first to draw attention to the role of model verification in model risk management In this excellent reference for model risk Morini discusses a scheme for model validation and notes the importance of model verification primarily as a check for internal consistency Although model verification and validation are used in managing model risk it is essential to treat both processes separately to ensure that a systematic approach can be taken to address model verification and validation Let s discuss the need for formal model verification next wilmott magazine

Page 12

A need for formal model verification in model risk management One might argue that model verification is easier in engineering and physical sciences where the laws of physics govern processes enabling the design of replicable processes based on test data In finance history rarely repeats itself and historical data provides limited value for verification Even worse for new innovations and custom financial engineering products such as structured products there isn t enough historical data to do proper model verification If this is the case what role does model verification play in finance for model risk management There are multiple reasons to advocate for a formal model verification step in the model review and validation process We list the main ones below Determining the degree of accuracy of implementation wilmott magazine business purpose and are conceptually sound and mathematically and statistically correct This essentially captures the essence of a formal model verification process Adherence to internal model risk and deployment policies In addition to regulatory requirements companies have internal model risk policies and guidelines for internal groups Many quantitative models developed in financial companies are proprietary and it is essential to vet them to ensure that companies aren t taking excessive risk by trusting models that aren t adequately tested and verified In addition to the technical requirements of a model quant groups are expected to implement business rules and organizational mandates into the model to There have been many regulatory efforts to advocate best practices for model risk management be used When legacy code is integrated with new code to build models it is important to ensure that the model functions as per design A formal model verification step helps to ensure this Adherence to regulatory requirements and guidance documents There have been many regulatory efforts to advocate best practices for model risk management The Dodd Frank Act Basel and the Solvency frameworks each address model risk and provide guidance on how institutions can incorporate model risk management programs The supervisory guidance on model risk management 5 is one of the key regulatory documents that provides extensive guidance on best practices and expectation of regulators when addressing model risk Although model verification isn t explicitly addressed many of the key aspects that model verification addresses are mentioned in this document A key statement in the section on Model Development and Implementation in the OCC document 5 notes Developers should ensure that the components work as intended are appropriate for the intended ensure that using the model adheres to internal policies Formal model verification can be used to check the adherence to various model risk policies adopted by the organization In addition when models are deployed into production they are modified to comply with information technology IT deployment policies These may include authentication and authorization rules logging transaction management exception handling etc In addition to software testing formal model verification can check adherence to IT deployment policies Finally it is widely acknowledged that quantitative models are complex and only a few people typically the model development group understand the intricacies of the model The model validation group can adopt model verification into the model validation process to verify the functionality of the models As the senior and supervisory management typically don t get to review the details of specific models incorporating formal model verification and setting expectations on clearance criteria at a policy level help senior management ensure that the model risk policies are implemented A model is typically made up of multiple components and dependencies In fact the supervisory guidance on model risk management 5 notes A model consists of three components an information input component which delivers assumptions and data to the model a processing component which transforms inputs into estimates and a reporting component which translates the estimates into useful business information In order to ensure that the model works as designed each component of the model should do its part and overall the model should perform as per design Having a formal model verification process facilitates verifying if each component and the model as a whole perform the role it was designed to perform This includes ensuring that the data and inputs are correct the processing component computes the requisite metrics as designed and the reporting or output component displays or stores results accurately In addition many models may not be fully developed in house and require data and code sourced from third party vendors and consultants Even if the model wasn t fully developed within the organization the impetus of ensuring that the model works as per design is still on the organization A formal model verification step acts as a gating process to ensure that the model can be deployed Even if the model s code wasn t sourced from outside many organizations reuse code built for other models requiring testing to verify if the reused code provides the desired functionality Also models are typically developed by multiple people and sometimes different groups A formal verification step can be used to verify if the integrated model works as desired A key challenge that organizations face when building models is dealing with legacy code When I worked for a large financial company I realized the challenges of working with legacy code when building models The project required using four libraries developed in three different languages over ten years by different groups and from an acquired company Companies which have gone through mergers and acquisitions inherit legacy code when companies and departments merge Sometimes the original developers are no longer available but models and functions continue to 57

Page 13

QuantUniversity Figure 1 The model verification process Scoping the Model Verification process Model Implementation Checks Understanding the art and science of model building Model development is both an art and a science and so is model verification The same model can be implemented in multiple ways using multiple algorithms and in different programming languages Model verification requires understanding the design intent and the implementation of the model and thoroughly testing if the model performs well both under normal and stressed conditions In addition model verifiers must creatively devise tests to verify if the model has weaknesses in handling edge cases or abnormal inputs Failure and error handling analysis i e whether a model fails gracefully with informative error messages or abruptly is another area that would require model verifiers to creatively devise tests that would cause models to fail and analyze failure modes Leveraging the best practices of model development and programming language specific guidelines a formal model verification step can uncover many issues that may not have been observed or encountered by model developers Elements of model verification Now that we understand the need for model verification in model risk management what constitutes model verification Having worked with many clients verifying their models we have developed best practices that we advocate to clients to help create and adopt a formal model verification program We have devised a four step model verification process that we believe every model verification process should have The four steps are summarized in Figure 1 and elaborated in this section Scoping the model verification process Prior to model verification it is essential that the 58 Model Policy and Process Checks Model Verification Reportiing scope of the model verification process be defined and that all stakeholders supervisors end users IT model developers model validators are aware of the process and the criteria used in the verification of the models Let s discuss this in detail Model scope We believe that the scope of model verification must include all the elements of the model that fall under the purview of model risk management The verification of the model should include the model source code and executables input and output files dependent libraries and execution environments that closely mirror how the model would be used We also advocate reviewing asso of model verification should include verification of all the components listed and maintained in the model BOM and the associated documents and artifacts like drawings workflows that detail how the various components interact in the design of the model Model specification model design model implementation Typically quant researchers prepare a formal model specification that describes the model in detail any modeling assumptions mathematics and the modeling approach The specification forms the basis for a design specification that details how the model would be implemented Model prototyping is typically done to demonstrate the feasibility and the characteristics of the final model The implementation team would then implement the production model in a chosen software platform Since the final model is the realization of the model specification and design model verification should include verification of the model along with the model s specification and associated design documents Note that the goal here is not to question the model s specification and approach but to verify if the model is implemented We advocate maintaining a model BOM that captures all the software components and the associated dependencies that are required for a model ciated documentation including design documentation methodology documentation and end user guides test plans and test results How can one systematically collect all the components required for model review In manufacturing a product assembly is typically associated with a bill of materials BOM The BOM provides a comprehensive list of all the components that go into the final product and helps to ensure that the finished product complies with the design We advocate maintaining a model BOM that captures all the software components and the associated dependencies that are required for a model The scope as per the specification and design Formal model validation is where the validity of the model to meet its intended purpose would be addressed Defining acceptance criteria To determine the correctness of implementation of the model it is essential to first determine acceptance criteria This helps communicate the intent and get agreement among stakeholders on the verification process It also helps to set the bar that needs to be achieved for the model to clear the model verification process Acceptable accuracy of the desired results and tests to be conducted magazine

Page 14

QuantUniversity needs to be defined prior to verification A review of model assumptions and approximations need to be done to determine what tests are feasible and appropriate for the model Performance criteria and service level agreements SLA need to be defined to ensure that the model meets design criteria and end user requirements It is recommended that the acceptance criteria be reviewed by the stakeholders prior to the model verification process to set appropriate expectations on this process Model implementation checks Model implementation verification is not trivial and is both an art and a science It requires expertise in programming and design and also an understanding of the mathematical model and modeling approach that were implemented As this is a complex task a formal model implementation verification plan needs to be developed after reviewing the model specification and design and understanding the model acceptance criteria Domain experts and implementation specialists should be consulted and the plan reviewed to ensure its completeness At a minimum the plan should include the following three elements as a part of the model implementation checks wilmott magazine of the hydraulic pump and are easily replaceable in the field We can draw parallels to input validation where checking for invalid inputs in the model before processing protects the model from potential model failures In models expected to do transactions output analysis is crucial Models must be A well designed model must anticipate potential issues and implement controls to address unanticipated failures appropriately Failure modes Error and exception handling is key to ensuring that a model is designed robustly and can be trusted to operate not only under normal conditions but also when a model encounters unanticipated conditions A well designed model must anticipate potential issues and implement controls to address unanticipated failures appropriately Model verification tests must check how models behave if invalid extreme inputs are presented When invalid inputs are presented tests must be conducted to evaluate if the models communicate intelligible error messages back to the users whether models log messages appropriately for replication and whether a model gracefully exits or crashes without messages Failure testing is as much an art as a science and isn t trivial Models often have multiple parameters and testing for all possible combinations of parameters is practically impossible Model verifiers must design tests to create failure conditions Input validation is a key part of the model ensures that invalid inputs are caught early and doesn t let the model continue if invalid inputs are presented This is a common practice in manufacturing and engineering I remember my experience as a mechanical engineer stress testing hydraulic pumps The input hoses connected to the pump were designed to fail early and protect the hydraulic pump The goal of stress testing was to ensure that when an abnormally high pressure invalid input was applied the hoses fail before the high pressure fluid reached the hydraulic pump Hoses are 1 100th of the cost checked for whether state management of variables has been done appropriately to ensure that in the event of model failures transactions are rolled back and invalid states cleared Tests must also be done to verify if appropriate logs and variable dump files are created to facilitate debugging and replication of issues Unanticipated model failures may occur for multiple reasons and it is impossible to forecast all potential failures But with adequate planning systematic testing for errors and ensuring that appropriate controls are implemented model failures can be minimized reducing the risk of catastrophic failures Determining the degree of correctness Checking if a model generates acceptable results is remarkably hard especially for complex financial products and for products that have limited historical data The model verification and acceptance criteria define the degree of correctness expected in the model s results One way to verify if the model generated results meet acceptance criteria is to benchmark results from the model with an independently created model in an alternate language platform These could be prototypes that quants have developed or alternative simplified models that generate results If a model is meant to replace an existing model backward compatibility tests can be conducted and the existing model can be used as the reference model In addition interactive debugging and code and variable introspection could be done to verify calculations Most programming The levers for the model Input output analysis Most models are designed to process inputs and generate certain outputs The input parameter space must be reviewed and tests designed to evaluate how the model performs for different values of input and output These tests should include normal and permissible input ranges and invalid and stressed inputs A review on whether the model performs as expected must be conducted and its impact assessed and documented Special attention must be paid to check input validation failure modes and controls to prevent unknown or unintended behaviors see section on Failure modes I have found that even basic what if analysis e g specifying invalid correlations invalid date calendar specifications etc can reveal problems and inconsistencies in models and when feasible I prefer to do thorough parameter sweeps to test how the model performs for a wide range of values A review of test plans and test results conducted by the model development team helps to determine the rigor of testing and if adequate testing was done during development Any issues found should be captured and reviewed to determine how critical the issues are and whether there are sufficient controls in place to address the risk posed by these issues refer to our paper on quantifying model risk 6 for additional details on this topic 59

Page 15

QuantUniversity environments provide variable and state inspection tools to facilitate incremental debugging Unit tests and functional tests enable the verification of atomic functionality in the models We also advocate reviewing the test plans and test results that model developers have run to verify the testing results and understand the rigor of the testing adopted during and after development verifying lists of outstanding issues and reviewing how well they are tracked categorized fixed and tested sheds light on the rigor of addressing model risk and checking for evidence of policy implementation This could be used to identify gaps in testing and verification and to introduce improvements into the model s life cycle management processes Model policy and process checks Once the model verification is completed the results must be appropriately documented and shared with the respective stakeholders for review We believe that a comprehensive report on model verification helps to assess the maturity of the model and whether it is truly ready for deployment Typically the model risk committee reviews the results against the acceptance criteria and deter Models not only are technical implementations of the model specifications but also encapsulate organizational business rules to adhere to the policies and controls adopted by the organization The model risk policy of an organization typically defines the regulatory and internal model policies that need to be followed when implementing mod Model verification reporting The model risk policy of an organization typically defines the regulatory and internal model policies that need to be followed when implementing models els This could include security and authentication rules data governance policies and how the model sources inputs and processes and saves results The model verification step should include a model policy check to ensure that the model implementation adheres to the published model risk policy In addition many organizations have IT and deployment guidelines to ensure that the models are ready for production The model verification step can be used to verify compliance with IT deployment policies and whether the models are indeed ready for primetime Finally as we noted in 2 as model risk drivers are permeated throughout the model s life cycle for effective model risk management model risk should be addressed throughout the life cycle The model verification step can be used to determine if the model implementation associated documentation test plans results etc comply with the documented policies for mitigating model risk throughout the model s life cycle For example if the organization has a policy on reviewing and addressing model issues through a formal process 60 validation processes We believe that formal model verification enables a systematic approach towards assessing and addressing model risk and assists in building a robust model risk management practice mines a plan of action to address issues that require attention prior to deployment After agreement from the model development team the plan of action should be documented and communicated to the stakeholders including senior and supervisory management Summary In this article we discussed the role of model verification in model risk management We differentiated model verification from model validation and discussed the need for a formal model verification process We then discussed four elements that we believe are essential components of a formal model verification process Many of the aspects of model verification that we discussed are meant to provide practical tips to operationalize the best practices of model risk management and to facilitate addressing many model risk concerns regarding model verification We encourage readers to consider customizing and adopting a formal model verification process as a part of their model review and References 1 Verification and validation of computer simulation models Available at http en wikipedia org wiki Verification_and_validation_of_computer_simulation_models 2 Krishnamurthy S 2014 The decalogue Wilmott July 58 61 3 DoD modeling and simulation M S verification validation and accreditation VV A 2009 DoD Instruction 5000 61 December 9 4 Morini M 2011 Understanding model risk In Understanding and Managing Model Risk Practical Guide for Quants Traders and Validators Wiley 5 Supervisory guidance on model risk management OCC 201112 SR 11 7 6 Krishnamurthy S 2014 Quantifying model risk Wilmott 69 56 59 About the Author Sri Krishnamurthy CFA CAP is the founder of www QuantUniversity com a data and quantitative analysis company QuantUniversity recently launched a new offering called Model Risk Analytics and offers advisory services for model risk management Sri has advised more than 25 customers providing asset management energy analytics risk management and trading solutions and teaches quantitative methods and analytics for MBA students at Babson College He is the author of several quantitative finance articles and is working on the forthcoming book published by Wiley titled Financial Application Development A case study approach He can be reached at sri quantuniversity com magazine

Model Risk Report

Page 1

Page 2

Page 3