Florian Resch
Analyst, Oesterreichische Nationalbank, Austria.
Wolfgang Aussenegg
Oesterreichische Nationalbank, Austria.
Gerhard Winkler
Oesterreichische Nationalbank, Austria.Pitfalls and Remedies in Testing the Calibration
Quality of Rating Systems
Testing calibration quality by means of backtesting is an integral
part in the validation of credit rating systems. Statistical methods are
used to assess if the ex-post realized default frequencies differ
significantly from the ex-ante estimate probabilities of default. Against
this background this paper provides a comprehensive overview of
existing testing procedures. We study the procedures' deficiencies
theoretically and illustrate their impact empirically. Based on the
insights gained thereof, we extend methods used for single rating
classes to develop joint tests for a rating system.
While exact tests require enumeration procedures and are
computationally expensive, approximate tests are fast to evaluate but
may yield strongly biased results. To combine the advantages of both
approaches we propose enhanced hybrid tests which rely on exact
Binomial distributions and approximations by Gaussian distributions.
Thus for the evaluation of the significance level we employ
enumeration and approximation techniques. Furthermore, we propose
computationally efficient algorithms for all multivariate tests. Finally,
we are able to demonstrate empirically that our method outperforms
existing tests in a scenario analysis using rating data of Moody's.
Hybrid testing procedures turn out to be superior to the commonly
applied methods in terms of speed versus accuracy.