Quality Money Management : Process Engineering and Best Practices for Systematic Trading and Investment

165

Check Performance and Shadow Trade

CHAPTER ◆ 17

In-sample and out-of-sample tests of a trading/investment system generate performance

metrics. Performance testing will validate the ability of the process to achieve the planned

results. Performance metrics should conform to those generated during prototype test-

ing. Trading system regression testing will detect errors. Nonconformance will cause the

product team to investigate its cause. Regression testing of a trading/investment system

seeks to uncover discrepancies between performance metrics the system generated during

a previous stage of development and those generated during backtests. These discrepan-

cies sometimes occur as a result of a change in tool sets, but at other times may be indica-

tive of the underlying nature of the trading system.

Develop

cleaning

algorithms

Perform

in-sample/

out-of-sample

tests

Gather

historical

data

Backtest

3

2

Check 1

performance

and shadow

trade

FIGURE 17-1

Experience shows that as trading/investment systems are developed and tool sets

change, regression bugs are quite common. Sometimes they occur because of poor con-

trol practices during the conversion from one tool set to another (e.g., conversion of Excel

prototypes into C algorithms, or a switch of optimization engines).

In most software development projects when a bug is located and fixed, good practice

dictates that the team run and record a test to prove the fix, and regularly retest the fix after

subsequent changes to the program. A test suite contains automated software tools that

allow the testing environment to execute all the regression test cases automatically. Some