268 CHAPTER ◆ 2 9 Determine Causes of Variation
● Political changes.
● Index composition changes.
● New, tradable contracts are offered.
● Better designed signals that reduce forecasting error.29.8. LOOP 1: Determine Causes of Variation in
Single Performance
Loop 1 should determine if the working system conforms to the backtest, and what are
currently normal results. Again, the purpose of a t -test or ANOVA is to find out whether
outputs from the backtest and the current system are the same. That is, to determine
whether the groups are actually different in a measured characteristic such as profit, win/
loss ratio, etc. In trading/investment systems, t -tests will determine if a system conforms
to its backtest, that is, are the performance metrics of the system from a common distribu-
tion as those of the backtest.
The question as to whether the system is working as designed or has the investment
environment changed in a manner that affects the model needs to be continually asked.
We also need to continuously seek out new ways to increase the forecastability of the
trading/investment strategy by removing explainable variation from the process. In many
systems the transformation of the system ’ s outputs can greatly decrease the variation.29.9. LOOP 2: Determine Causes of Variation in
Attribution
A factorial design is used to evaluate two or more factors simultaneously. The advantages
of factorial designs over one-factor-at-a-time experiments are that they are more efficient
and they allow interactions to be detected. After the attribution analysis is completed and
we understand all of the bets we are placing to beat the benchmark, we need to build one
final set of tools. These tools are based on statistical process control—namely, design of
experiments and ANOVA.
The goal of these tools is to determine the causes of variance in a portfolio ’ s returns
and increase the excess returns. If you accept that the best you can do is beat your bench-
mark by several 100 basis points by placing better bets then you should also accept the
fact that the majority of the variance in your portfolio is driven by macro market condi-
tions. Therefore, by applying SPC to monitor your portfolio selection algorithm and the
benchmark you can hopefully determine when the algorithms are not working properly.
During the times when the algorithms do not work properly you should either scale down
your bets or revert back to benchmark.John Brush showed that the difference between other published studies and his studies was that he
deliberately removed data from Decembers and Januarys from his result set. He claims that is why most
articles showed that systems did not perform well and his results showed that they could. This can be
described as a clean application of ANOVA between months and a group of t -tests to identify the months
that have different means, or processes. Brush showed an algorithm that worked versus classical finance
academics who did not perform ANOVA and stated that the algorithm did not work.