Backtesting: Deeper investigation of specific portfolio segments

Backtesting the allowance for loan and lease losses (ALLL) enables banks and credit unions to compare actual results for a defined period to the results forecasted by a model for the same period in order to evaluate accuracy of the model’s predictiveness.

When evaluating specific portfolio segments, it is important to determine answers to several questions from period to period:

• Which segments had an increase in the FAS 5 reserve?

• How much of the increase is due to changes in volume of that portfolio? 

• How much is due to change in loss rate?

After identifying that some of the reserve change is due to loss rate changes, the institution must then analyze how much of the effect is due to a change in the historical loss rate rather than the applied qualitative and environmental factors.

When looking at the historical loss rate component, the institution must examine the timing of charge-offs that rolled out of the loss rate as compared to charge-offs that entered in with the most recent quarter. These changes can be anticipated and/ or projected, so it is important to examine this closely ahead of time.

If the institution is utilizing migration analysis, then it is important to compare the numerator and denominator of the loss rate for each segment and sub-segment; it may indicate that deterioration of a portfolio led to a higher rate in one sub-segment than in another. For example, if the loss rate is now higher for substandard, this is either because:

1. In the source historical period, the volume of substandard loans is lower

2. The level of charge-offs attributable to substandard loans is now higher.

These types of differences can be discovered by a report similar to below.

In the example above using an 8-quarter migration period starting in 9/2011 and 6/2011 for measuring loss rates in 2Q13 and 3Q13, the loss rate for Pass Rated loans in the CRE portfolio has decreased fairly significantly, as it appears some charge-offs dropped outof the migration period used during this time period. The Special Mention has a slight increase in loss ratio applied for the CRE portfolio, due to a smaller Special Mention portfolio in the denominator of the loss rate for 3Q2013 (using 9/2011).

When looking at the qualitative factor aspect, a bank should look closely at the change in qualitative factors over time and compare the direction of change to the direction of change of any applicable metrics for that specific factor. Identification of key drivers and directional consistency are important in measuring the effectiveness of a bank’s qualitative and environmental factor methodology.

Also, backtesting of the realized charge-offs as compared to the adjusted historical loss rate should be completed. This would require looking at an adjusted historical loss rate at a set point in time and then examining the amount of net charge-offs over the subsequent 12 months.

Overall, the importance of backtesting is growing as a useful way to defend a bank’s ALLL methodology. To learn more about best practices, including specific examples, download the whitepaper, Backtesting: Measuring the Effectiveness of ALLL Methodologies.