206 MODEL RISK AND FINANCIAL DERIVATIVESIt is important to realize that the role of model testing should not be
reduced to validate or invalidate a model, but should also include increasing
its reliability, revealing its weaknesses, confirming its strengths and pro-
moting improvements. Consequently, it is essential that (1) any information
generated during the test phase be recorded and documented; and (2) pur-
chasing a model from an external vendor does not exempt it from the
validation process.
Rule 4: Regularly challenge and revise your models
At the root of the model risk problem is that market and mathematical
assumptions (for example, simplifications of market behavior) are often
hard-coded and remain stagnant within the model, while things do change
in real life. Consequently, models should not be carved in stone, but rather
evolve and improve with time. All models used within an institution should
be regularly revised and their adequacy to the current market conditions
challenged. This process should include an analysis of the underlying
assumptions as well as a consistency check with the best-accepted practices
in the industry. In addition to this regular revision, institutions extending
existing businesses or entering new ones should also make a special effort
to reassess existing models, procedures, data and best practices before they
adopt. Very naturally, model users should be involved in the process as they
are likely to be aware of the latest developments in the field.
Rule 5: Mark to market or to market standards, not to a model
Following the Group of Thirty’s (G30) recommendations, the calculation
of the mark-to-market value^7 of derivative positions is widely practiced in
the financial industry as a natural way to avoid model risk. Unfortunately,
marking to market has its own dangers and may induce a false sentiment of
security and overconfidence.
For positions where there is a conventional market price (for example,
closing bid), one would expect the results of a good model to be quite close
to those observable on the market. Appreciable differences should be seen as
an early warning system, so that one needs to fully understand the sources
of these differences to form an opinion of the model being tested. As an
illustration, the 1997 disaster at NatWest could have been easily avoided by
obtaining external implied volatility quotes from brokers or other institu-
tions that trade in the marketplace and by comparing them with Natwest’s
values.
For more complex or illiquid derivative instruments, marking to market
becomes a difficult exercise. When prices are not easily available, traders
tend to use theoretical prices as a benchmark, generating an important