Building and Testing a Multiple Linear Regression Model 85
which independent variables to use. Increasing the number of independent
variables does not always improve regressions. The econometric theorem
known as Pyrrho’s lemma relates to the number of independent variables.^4
Pyrrho’s lemma states that by adding one special independent variable to
a linear regression, it is possible to arbitrarily change the size and sign of
regression coefficients as well as to obtain an arbitrary goodness-of-fit. This
tells us that if we add independent variables without a proper design and
testing methodology, we risk obtaining spurious results.
The implications are especially important for those financial models
that seek to forecast prices, returns, or rates based on regressions over eco-
nomic or fundamental variables. With modern computers, by trial and error,
one might find a complex structure of regressions that give very good results
in-sample but have no real forecasting power.
There are three methods that are used for the purpose of determining
the suitable independent variables to be included in a final regression model.
They are:
- Stepwise inclusion regression method
- Stepwise exclusion regression method
- Standard stepwise regression method
We explain each next.
Stepwise Inclusion Regression Method
In the stepwise inclusion regression method we begin by selecting a single
independent variable. It should be the one most highly correlated (positive
or negative) with the dependent variable.^5 After inclusion of this indepen-
dent variable, we perform an F-test to determine whether this independent
variable is significant for the regression. If not, then there will be no indepen-
dent variable from the set of possible choices that will significantly explain
the variation in the dependent variable y. Thus, we will have to look for a
different set of variables.
If, on the other hand, this independent variable, say x 1 , is significant,
we retain x 1 and consider the next independent variable that best explains
the remaining variation in y. We require that this additional independent
variable, say x 2 , be the one with the highest coefficient of partial determina-
tion. This is a measure of the goodness-of-fit given that the first x 1 is already
in the regression. It is defined to be the ratio of the remaining variation
(^4) T. K. Dijkstra, “Pyrrho’s Lemma, or Have it Your Way,” Metrica 42 (1995): 119–225.
(^5) The absolute value of the correlation coefficient should be used since we are only
interested in the extent of linear dependence, not the direction.