Formulating and Implementing Investment Strategies Using Financial Econometrics 309
the first step toward recovery is the recognition of the problem. The next
step is to facilitate a research environment that avoids the temptations of
making choices.^3 What this conclusion also means is that researchers should
be extremely disciplined at every step of the process in making choices, with
the explicit understanding of the bias induced from data snooping.
Select a Sample Free from Survivorship Bias
Since all backtest research is performed on a data set that looks back in
time, the entire history of an observation will not be available if it does
not survive the present. The sample that researchers can work with is a set
of observations that have been preselected through time by some common
denominators. A sample of a sample should not pose a problem if the subset
is selected randomly. But this is not the case for most samples, which suffer
from survivorship bias as we described in the previous chapter. The bias
becomes relevant if the common bond to survive the observation is related
to the pattern for which we are looking. A finding of a statistically signifi-
cant pattern merely reflects the underlying common bond that was used to
construct the testing sample.
One typical point of interest, which is severely affected by the survivor-
ship bias, is performance comparison. By only looking at the portfolios cur-
rently outstanding, it is obvious that portfolios that did not survive through
time due to poor performance are excluded from the sample. By design, the
sample only contains good portfolios. How can the true factors that have
caused bad performance ever be identified?
Commercial data vendors are not helping on this issue. Due to cost
consideration, most data sets are only provided on a live basis. That is, for a
currently non existent sample observation, the common practice is to delete
its entire history from the data set. To simulate the true historical situation,
it is the researchers’ responsibility to bring these observations back to the
sample. The sample collection procedure should be reversed in time. Cases
that existed at the beginning of the sample period should be included and
tracked through time.
Select a Methodology to estimate the Model
The selection of a certain methodology should pass the same quality tests
as developing financial economic theories and selecting samples. Without
strong intuition, researchers should choose the methodology that needs the
(^3) This reluctance is probably the single most important reason for the recently devel-
oped machine learning techniques discussed in the previous chapter.