132 CHAPTER ◆ 1 3 STAGE 2: Overview
Because backtests have multiple objective functions, the process leaves the field of
pure mathematics, where environmental factors must be static, and enters what we call
the “ field of NASCAR, ” where a team attempts to fine-tune the machine for optimal per-
formance in the current environment in order to beat the competition. Quality should not
be understood as an attempt to hit a performance target. Quality is not mathematics or an
optimization routine. Instead, the product team uses quality techniques to minimize varia-
tion in the process, and optimization to maximize returns. Our methodology blends these
two ideas together, using heuristics and design of experiments (DoE), to understand the
interactions between multiple factors and the environment and the effects of those interac-
tions on the stability of the trading/investment machine. The goals of backtesting are to:● Determine the likely profitability of the trading/investment system.
● Understand the risks of the system.
● Understand the common variation inherent in the system, that is, the stability of the
system.
● Measure the effects of economic cycles on the trading/investment algorithms.As Reiner Bohlen reports, “ Effective backtesting requires use of many resources, includ-
ing data streams, data vendors, software vendors, application developers, hardware plat-
forms, and personnel... Instead of requiring a human pilot to manually manipulate data
and optimize every month of a backtest, organizations create integrated software environ-
ments that reflect their unique business processes and technology. ”^2
While backtesting will reveal the performance characteristics of the system, the primary
issue at hand is the need to accurately produce multiple trials that, time and again, perform
exactly the same, or very nearly so. Backtests should be executed ideally in a high-level,
backtesting language which often will be provided by a third-party vendor (MarketQA
language, Zack ’ s Backtest, Quantifi, Sungard ’ s FAME, Factset, etc.). Excel, Resolver,
MATLAB, and SAS can also be used to backtest effectively; however, the team should take
care to lock down scripts and workbooks for zero user-input or command line entry.
In some instances, the product team may be able to or be required to convert and
compile all of the calculations in C, Java, or .NET code objects that can be used in
the production system. By coding algorithms, the team can be sure they will be iden-
tical in future tests. Human interaction at this stage, where team members can and do
alter the internal structure of the system for each test, promotes errors. It must be avoided
and the system must be rigid, which is why we recommend producing an investable uni-
verse. Where algorithms are converted to code, the team should not spend extra time and
resources on programming at this stage, while being careful not expand the scope of pro-
gramming beyond that of components absolutely necessary for backtesting.
Resolver software is a particularly powerful tool because it allows developers to eas-
ily create spreadsheets that integrate databases, Python code, and external components
such as real-time data APIs. A Resolver application is at the same time a spreadsheet and
a Python computer program. Spreadsheet algorithms are coded automatically in Python,
which makes the conversion step, at least into Python, irrelevant.13.1. Converting Prototypes to Coded Models
Relative to the Research and Document Calculations stage, K|V Stage 1, backtesting may
require a tool change from prototypes (in Excel, Resolver, MATLAB, SAS, etc.) to coded
implementation of trading algorithms. In such cases, regression testing is key to achieving
successful and reliable development of the software that implements the trading/investment