12-FinEcon-Model Sel Page 318 Wednesday, February 4, 2004 12:59 PM
318 The Mathematics of Financial Modeling and Investment Managementselecting the right model complexity. The complexity of a model is
sometimes identified with its dimensionality, that is, with the number of
free parameters of the model.
The problem of model complexity is intimately connected with the
concept of algorithmic compressibility introduced in the 1960s indepen-
dently by Andrei Kolmogorov^4 and Gregory Chaitin.^5 In intuitive terms,
algorithmic complexity is defined as the minimum length of a program
able to reproduce a given stream of data. If the minimum length of a
program able to generate the given sequence is the same as the length of
the data stream, then there is no algorithmic compressibility and data
can be considered purely random. If, on the other hand, a short pro-
gram is able to describe a long stream of data, then the level of algorith-
mic compressibility is high and scientific explanation is possible.
Models can only describe algorithmically compressible data. In a
nutshell, the problem of learning is to find the right match between the
algorithmic compressibility of the data and the dimensionality of the
model. In practice, it is a question of implementing a trade-off between
the accuracy of the estimate and the size of the sample.
Various methodologies have been proposed. Some early proposals are
empirical rules of thumb, based on increasing the model complexity until
there is no more gain in the forecasting accuracy of the model. These pro-
cedures require partitioning the data in training and test sets, so that
models can be estimated on the training data and tested on the test data.
Procedures such as the Box-Jenkins methodology for the determina-
tion of the right ARMA model can be considered ad hoc methods based
on specific characteristics of the model, for instance, the decay of the
autocorrelation function in the case of ARMA models.
More general criteria for model complexity are based on results
from information theory. The Akaike Information Criteria (AIC) pro-
posed by Akaike^6 is a model selection criterion based on the informa-
tion content of the model. The Bayesian Information Criteria (BIC)
proposed by Schwartz^7 is another model selection criterion based on
information theory in a Bayesian context.(^4) Andrei N. Kolmogorov, “Three Approaches to the Quantitative Definition of In-
formation,” Problems of Information Transmission 1 (1965), pp. 1–7.
(^5) Gregory J. Chaitin, “On the Length of Programs for Computing Finite Binary Sequenc-
es,” Journal of Association Computational Mathematics 13 (1965), pp. 547–569.
(^6) H. Akaike, “Information Theory and an Extension of the Maximum Likelihood
Principle,” in B.N. Petrov and F. Csake (eds.), Second International Symposium on
Information Theory (Budapest: Akademiai Kiado, 1973), pp. 267–281.
(^7) Gideon Schwarz, “Estimating the Dimension of a Model,” Annals of Statistics 6
(1978), pp. 461–464.