204 MODEL RISK AND FINANCIAL DERIVATIVESRule 1: Define what should be a good model
Before qualifying a model as being better than another model, one needs to
define precisely the model-quality metrics. What makes a model superior to
another often varies according to its usage. Consider, for instance, a pricing
model versus a risk management model. The former must match closely
the values of known liquid securities and focuses on absolute prices today,
while the latter needs to realistically represent the possible future evolution
of market variables and focuses on relative variations tomorrow. Both goals
are sufficiently different to result in divergences in the ranking of competing
models, depending on the context. Therefore, a good pricing model will not
necessarily be a good risk-management model, and vice-versa. Nor will
the hypothesis or conclusions reached for the pricing model be necessarily
applicable to the hedging model.
In addition to the model usage, what makes a model superior to another
also varies according to the preferences of the model user and the necessary
underlying assumptions. Unfortunately, practitioners often understate or
neglect this aspect. For instance, it is common practice to test an option pric-
ing model’s quality by comparing the model’s predicted prices with market
prices, and using some sort of loss function such as the mean pricing error
(with respect to the market), the mean absolute pricing error or the mean
squared pricing error. The approach implicitly assumes that (i) model users
display symmetric preferences (for example, equally consider under- and
over-pricing, or care equally about losses and gains); (ii) the option pric-
ing model is correct; and (iii) the market is efficient in its pricing process.
If the option pricing model is rejected, we do not know if (ii) is untrue;
(iii) is untrue; or (ii) and (iii) are untrue! And even if the model has been val-
idated, the validation process may not hold true for an individual displaying
asymmetricpreferences(forexample, downsideriskaversion). Itistherefore
crucial to agree on the model-quality metrics prior to ranking any type of
model.
Rule 2: Keep track of the models in use
Fighting model risk should start with a detailed inventory of the various
models available within a financial institution. This means keeping records
of which models are used, who uses them, and how they’re used. For
computer-based models, it also implies keeping track of who built them,
who keeps the code and who is allowed to change it. It is not uncommon to
see banks where the only version of the source code is stored on a magnetic
tape somewhere in the archives.
An up-to-date inventory should significantly enhance productivity and
reliability. Typically, in a financial institution, each time a new product is