266 CHAPTER ◆ 2 9 Determine Causes of Variation
A secondary result of using Pareto analysis is that the largest sources of variation are
attacked first. Through the continuous reduction in variation we can focus on forecasting
to produce higher alpha indicators.29.5. Design of Experiments
Six Sigma ’ s five step DMAIC—Define, Measure, Analyze, Improve, Control—is a data-
driven quality methodology for improving a process. To analyze and subsequently improve
a process, Six Sigma uses design of experiments (DoE) combined with analysis of variance
(ANOVA).
Design of experiments is a structured, organized approach to measuring the relation-
ship and interactions between factors, that is, controllable independent variables, that affect a
process and the output of that process. DoE has been successfully applied to many differ-
ent types of process including the standard mixture model products. In financial markets
it is difficult, if not impossible, to reproduce financial market data and trading/investment
system performance exactly. Comparisons between data sets is much more reproducible.
This is why the product team should perform experiments between the backtest results
and the index benchmark as a baseline.
By comparing results, analysis of variance breaks down total variation into components.
In financial markets, we sometimes call these components noise and signals. There is a sig-
nal component for each controlled variation and a noise component representing variations
not attributable to any of the controlled variations. By looking at the signal-to-noise ratio for
a particular variation, analysis of variance will provide clues and answers about root causes.
In trading/investment system management, risk managers are confronted by signifi-
cant degrees of uncontrollable variation in data and environmental conditions. Such prob-
lems can be overcome by running properly constructed experiments. Fortunately, the ease
of capturing market data and trading system performance make meaningful experiments
possible, if systems for performance analysis have been built in over the stages of devel-
opment and testing. Modern statistical software packages will perform the appropriate
analysis of variance at the click of a mouse.
Design of experiments depends upon the selection of appropriate factors and
responses, inputs and outputs, factor ranges, and as always documentation of experiments
and discoveries. We believe that interaction between trading/investment signals should be
perceived within the framework of design of experiments. There are then three bets a sys-
tem may make (and this applies especially to filter and multifactor systems):● Getting the sector right, which may, for example, be best implemented by buying ETFs.
● Getting the individual stocks right, which forces an equal weight or an underweighting
of the sector versus the index.
● Getting the interaction right, where ideally the system picks both the sector right
and the stocks right.Our view of interaction and its importance is common in industrial statistics, where the
use of design of experiments to tune industrial processes is common. Perhaps the most
interesting fact is that many of the primary contributors to the development of DoE—
Box, Cox, Jenkins, Hunter, and Montgomery—did so toward the end of their long,
prestigious careers in industrial statistics, which included invention of many standard
techniques for time series analysis currently used in finance.