Quality Money Management : Process Engineering and Best Practices for Systematic Trading and Investment

167

Finally, the team should decide on the tools, tables, and graphs used to mathemati-

cally validate the results of the in-/out-of-sample testing. We recommend that at a mini-

mum the product team determine the types of graphs that will be produced as part of the

check performance stage. These tools will be used in loops 2 and 3 and will form the

basis of statistical control tools.

17.2. STEP 4, LOOP 2: Perform Regression Test of

In-Sample Results against Prototype

In this step, the product team performs a regression test of the in-sample results versus

the investable universe and versus the benchmark in order to prove that the system per-

forms as expected.

The quality of the trading/investment system will be appraised by a collection of

regression tests that evaluate one or more of its features. A valid regression test of in-

sample results will generate the verified gold standard results. Regression tests data and

the gold standard results should be packaged into a regression test suite. As development

progresses and more tests are added, new results from in-sample and out-of-sample tests

can be compared with the gold standard results. If they differ, then a potential flaw has

been found in the system.

Depending on the nature of the flaw, it may either be corrected and development

allowed to continue, or development will revert back to Stage 1. This mechanism detects

when new tests invalidate existing strategies and thus prevents the trading/investment sys-

tem from progressing any further.

17.3. STEP 4, LOOP 3: Perform Regression Test of

OS Test Results against IS Results and

SPC Outputs and Shadow Trade

In this step, the product team performs regression testing of all the performance metrics

against the in-sample, gold standard test results. The performance metrics should be simi-

lar and the degree of similarity will indicate if the environment is currently favorable for

this algorithm. Let us suppose, however, that you produce an algorithm that works well

in sample, but that has a clear economic cycle to it. You run the out-of-sample test, and

you see that the economic cycle is becoming unfavorable; the out-of-sample results show

a shift in performance versus the in-sample results. This could lead to a parking of the

system at the next stage gate or at least a frank discussion of risk with management. The

reverse situation is that the environment is becoming more favorable for the system, which

should prompt management to release resources more quickly to bring the system online.

At this stage, the team should now know the key inputs, the key factors, and perform-

ance metrics to monitor, and have graphs showing excess returns by week or by day.

Now, the team can introduce SPC graphs; at a minimum a range graph showing upper

and lower control limits, due to the volatility of the indicators. Included should be C pk

calculations between the in-sample and out-of-sample tests, and if applicable perform-

ance specification limits. The team should also consider control charts, using, for exam-

ple, a moving average of performance metrics. Management should expect all of these

visual performance indicators at the Gate 2 meeting to show the stability of the system

17.3. STEP 4, LOOP 3: PERFORM REGRESSION TEST