12-FinEcon-Model Sel Page 317 Wednesday, February 4, 2004 12:59 PM
Financial Econometrics: Model Selection, Estimation, and Testing 317For instance, an econometrician might decide, on theoretical grounds, to
adopt an ARMA family of models. Searches will then help determine
parameters such as the order of the model and the estimation of the
model parameter. We will return to the problem of determining the
model complexity and estimating parameters in the following sections.
The above considerations apply to parametric models, that is, mod-
els that include parameters to be estimated. There are statistical models
that appear to be nonparametric. Nonparametric models are typically
based on the empirical estimation of probability distribution functions.
Nonparametric models are typically simple models as there is no practi-
cal way to estimate empirically complex models.
In summary, econometrics follows a general scientific principle of
formulation and testing of theoretical hypotheses. However, economet-
ric hypotheses are generally formulated as a family of models with
parameters to be optimized. Econometrics is thus an instance of a gen-
eral process of learning.^3LEARNING AND MODEL COMPLEXITY
If one had an infinite amount of empirical data and an infinite amount of
computational resources, econometric models could in principle be selected
with arbitrary accuracy. However as empirical data are finite and, gener-
ally, scarce, many different models fit empirical data. The key problem of
statistical learning is that most families of models can be parameterized so
that they can fit a finite sample of data with arbitrary accuracy. For
instance, if an arbitrary number of lags is allowed, an ARMA model can be
made to fit any sample of data with arbitrary accuracy. A model of this
type, however, would have very poor forecasting ability. The phenomenon
of fitting sample data with excessive accuracy is called overfitting.
In the classical formulation of the physical sciences, overfitting is a
nonissue as models are determined with theoretical considerations and
are not adaptively fit to data. The problem of overfitting arises in con-
nection with broad families of models that are able to fit any set of data
with arbitrary accuracy. Avoiding overfitting is essentially a problem of(^3) Christian Gourieroux and Alain Monfort, Statistics and Econometric Models
(Cambridge: Cambridge University Press, 1995); D.F. Hendry, “Econometrics: Al-
chemy or Science?” Economica 47 (1980), pp. 387–406, reprinted in D.F. Hendry,
Econometrics: Alchemy or Science? (Oxford: Blackwell Publishers, 1993, and Ox-
ford University Press, 2000); D.F. Hendry, Dynamic Econometrics (Oxford: Oxford
University Press, 1995); and Vladimir N. Vapnik, Statistical Learning Theory (New
York: John Wiley and Sons, 1998).