Quality Money Management : Process Engineering and Best Practices for Systematic Trading and Investment

139

Gather Historical Data

Once the initial prototype has shown a system worthy of further investment of time

and resources, the real make-or-break task of backtesting begins, where the team truly

tests the ability of the trading/investment system to meet the specifications set forth in

the Money Document. Prior to building and implementing the system, a product team

tests it over a large sample of historical data, and in many cases for a large sample of

instruments. Backtesting should reveal if the performance results experienced in Stage 1

are real and scalable or due simply to overfitting the original data sample. Backtesting

requires lots of data versus the Stage 1 tests, which used small sets of data. Product teams

purchase customized databases of past data and purchase software systems that facilitate

the backtesting process, which in itself can create problems due to known and unknown

errors in the data and software that may cause variation not seen in the sample data.

While in theory buying data and software appears simple, properly integrating systems

is complex and time-consuming due in part to a lack of understanding of how to build a

scaled prototype test environment.

CHAPTER ◆ 14

FIGURE 14-1

1

2

3

Develop

cleaning

algorithms

Perform

in-sample/

out-of-sample

tests

Check

performance

and shadow

trade

Gather

historical

data

Backtest

Since the necessary data may either not exist at all or may be prohibitively expensive

based upon the prospective returns of the trading/investment system, investigating the

availability and price of data ahead of time is a very important first step.