Model Selection 299
how can it be tested? These are complex questions that do not admit an
easy answer. It is often assumed that financial markets undergo “structural
breaks” or “regime shifts” (i.e., that financial markets undergo discrete
changes at fixed or random time points).
If financial markets are indeed subject to breaks or shifts and the time
between breaks is long, models would perform well for a while and then,
at the point of the break, performance would degrade until a new model
is learned. If regime changes are frequent and the interval between the
changes short, one could use a model that includes the changes. The result
is typically a nonlinear model. Estimating models of this type is very oner-
ous given the nonlinearities inherent in the model and the long training
period required.
There is, however, another possibility that is common in modeling.
Consider a model that has a defined structure, for example a linear vec-
tor autoregressive (VAR) model (see Chapter 9), but whose coefficients are
allowed to change in time with the moving of the training window. In prac-
tice, most models work in this way as they are periodically recalibrated. The
rationale of this strategy is that models are assumed to be approximate and
sufficiently stable for only short periods of time. Clearly there is a trade-off
between the advantage of using long training sets and the disadvantage that
a long training set includes too much change.
Intuitively, if model coefficients change rapidly, this means that the
model coefficients are noisy and do not carry genuine information. There-
fore, it is not sufficient to simply reestimate the model: one must determine
how to separate the noise from the information in the coefficients. For exam-
ple, a large VAR model used to represent prices or returns will generally be
unstable. It would not make sense to reestimate the model frequently; one
should first reduce model dimensionality with, for example, factor analy-
sis (see Chapter 12). Once model dimensionality has been reduced, coef-
ficients should change slowly. If they continue to change rapidly, the model
structure cannot be considered appropriate. One might, for example, have
ignored fat tails or essential nonlinearities.
How can we quantitatively estimate an acceptable rate of change for
model coefficients? Are we introducing a special form of data snooping in
calibrating the training window?
Calibrating a training window is clearly an empirical question. How-
ever, it is easy to see that calibration can introduce a subtle form of data
snooping. Suppose a rather long set of time series is given, say six to eight
years, and that one selects a family of models to capture using financial
econometrics the DGP of the series and to build an investment strategy.
Testing the strategy calls for calibrating a moving window. Different moving
windows are tested. Even if training and test data are kept separate so that