Model Risk Management in the Time of COVID-19

The Coronavirus (COVID-19) crisis has imposed an enormous toll on the global economy, significantly hindering international trade and hurting numerous businesses and consumers. The crisis highlighted the importance of evaluating risk from all angles and maintaining a coherent risk management program at financial institutions.  

In this alert, Guidehouse discusses the implications of the COVID-19 pandemic on bank risk management models and proposes a few tips to master model recalibration. Banks may find it necessary to take an exhaustive look at their internal and external risk management models and capitalize on knowledge gained through previous crises.

Black swan events are inherently unpredictable and cause sudden and major breaks in econometric and statistical models. Such breaks manifest themselves as significant shifts in economic scenarios and the underlying distribution of parameters. There is a consensus around the fact that both traditional and machine learning/artificial intelligence (ML/AI) models’ predictive power depends on the quality of their data and the set of assumptions regarding the underlying data-generating mechanism. For a robust modeling approach, it is imperative that model failures be quickly identified, and remediation plans be developed and implemented in response to such failures.

In the event of model failures, banks should identify assumptions that no longer hold, revise scenarios, and rerun stress tests under updated assumptions for both short-term and long-term behavior of macroeconomic variables. Accordingly, Randal Quarles, the Federal Reserve’s vice chairman for supervision, stated in a recent discussion that existing hypothetical adverse scenarios do not sufficiently reflect current events, and stress tests would incorporate changes in the parameters due to the pandemic.   

The economic crisis triggered by the pandemic created shocks in major macroeconomic variables:

The simultaneous increase in variance and fat tails in the distribution of input variables may impact the stability of the relationship between dependent and independent variables. Out-of-sample, or out-of-time validation is typically not reliable when there are substantial differences in the training, validation, and testing partitions of the input data. In crisis periods, ML/AI models do not outperform conventional statistical models, as they are, by design, incapable of identifying biases in the data. Turning vast amounts of data into business intelligence is not an easy task, and model risk typically increases with unvalidated assumptions and inputs. 

Adverse scenarios, which are typically informed by historical data, may not fully capture the spectrum of all plausible events, specifically those that are unprecedented or that have remote probabilities of occurring (fat tails). On a related note, reliance on historical data alone may misinform models on the pace of deterioration on loan portfolios, and thus cause a mismatch between the predicted and realized time of events. Similarly, the accumulated debt over the pandemic could potentially be a drag on the loan recovery rates if there is a change in consumers’ appetite for risk that can eventually lead to reduced investments, and consumption behavior, which can impede spending.

It would not be surprising if the model pass/fail thresholds were breached more often in crisis times than normal times. It is widely documented that the stochastic process governing loan portfolio performance differs across various regimes. This structural break leads to larger errors and poor model performance statistics. When models’ predictive power deteriorates, model owners should strive to understand the source of such deterioration, identify the factors — e.g., those that are liquidity-related versus credit-related — that lead to larger errors and recalibrate using managerial judgment (overlays). Some recalibration may be necessary to include more recent data points. However, given the lack of sufficient data, as the crisis is still developing and not fully reflected on the macro variables, reliance on overlays may be the only feasible route. Data from a previous crisis may appear to be the most likely candidate to inform management/model owners about the overlays. Yet, even such data could be a poor benchmark for the future, because, as mentioned above, the data-generating process may vary across crisis periods.

Risk managers need to thoroughly assess the impact of the crisis on their client base and evaluate portfolio segments based on vulnerability. Accordingly, model assumptions may need to be updated to accommodate the varying expectations across retail and institutional loan portfolios. Moreover, credit models (e.g., Current Expected Credit Losses) and liquidity models should be reevaluated to incorporate the distributional characteristics of sectoral and geographical exposures. The impact of shocks on some sectors, such as retail and hospitality, may potentially be larger, at least in the short run, than the impact on some other sectors. Similarly, some Metropolitan Statistical Areas may be hit harder than others. Therefore, loans originated in these areas may have more variation in default and recovery probabilities. In a similar fashion, deposit models should be reassessed under various interest regimes, of which some could potentially be prolonged for longer than historical averages.  

While leveraging data and analytics could potentially provide useful insights, management/model owners would have to weigh in with institutional knowledge to quantify overlays. This process inevitably entails, to a large extent, relying on subjective judgement. Nonetheless, the onus still falls on the model owner to measure the overlays against benchmarks. A healthy skepticism should be embedded in the implementation of overlays.

Achieving good model performance should be more intentional than accidental. Therefore, it is vital to develop a framework and follow through with a set of questions as a stepping stone to contemplate a robust model recalibration strategy:

  • What data sources are available to inform the construction of overlays?
  • Has management evaluated alternative overlay values? 
  • Do overlays mitigate or introduce any biases in the data? 
  • Do overlays potentially violate any regulations around Fair Lending laws, the Community Reinvestment Act, the Home Mortgage Disclosure Act, etc.?
  • Are stakeholders comfortable with implementation of the model with the overlays? 
  • How frequently do overlays need to be reassessed for accuracy and relevancy? 

Applying overlays is not a silver bullet for model recalibration. Management/model owners should also consider the fact that seemingly uncorrelated factors in normal times may become highly correlated in crisis periods, which can add another layer of complexity in model risk management. This is expected when the root of the crisis is outside the financial ecosystem but presents itself as an insurmountable challenge for the whole economy. The remedies implemented by regulators are not necessarily always directed at the root, but at the symptoms, of the crises. Hence, the response of the macroeconomic variables to remediation plans may be delayed and this may potentially create model performance issues in risk management. 

In conclusion, the COVID-19 pandemic has brought about profound changes to interactions between consumers and businesses. Model risk management is not resilient to rapidly evolving consumer and investor trends. We recommend that banks take the following five steps to address these unprecedented changes:

  1. Test for model failures and enhance their model performance monitoring procedures. 
  2. In the event of model failures, test whether assumptions still hold, revise and revalidate scenarios, and rerun stress tests under updated assumptions.
  3. Risk management should compare the impact of the crisis across portfolio segments with varying vulnerability.
  4. Banks pursuing model recalibration should develop a framework and consider remapping sources of risk and model parameters under the current paradigm.
  5. Management/model owners need to leverage institutional knowledge in the development and implementation of risk overlays.

Banks that do not revisit their risk models and consider model recalibration face the risk of creating a mismatch between their risk exposures and future actions. This mismatch could potentially cause significant gaps in decision-making and could result in portfolio delinquencies far beyond management’s predictions. Risk models that exploit data under the new paradigm, accompanied with qualitative adjustments, could arm banks with a solid tool in crisis management. Banks that recognize the pitfalls of the status quo are likely to benefit from better alignment between their risk exposures and target portfolio risk. 

About the Experts

Back to top