Quality Money Management : Process Engineering and Best Practices for Systematic Trading and Investment

135

for the full-scale experiment. The goal is to create a stable test environment, defined as

a system where all assignable causes are removed. Also, the team will remove known

errors, and produce the data cleaning algorithms.

13.4. LOOP 2: Optimize Signals

STEP 1: Purchase Data

STEP 2: Clean and Adjust for Known Issues

STEP 3: Perform In-Sample Test for Large Sample

STEP 4: Perform Regression Test of In-Sample Results against Prototype

Loop 2 is the actual full-scale experiment. From an industrial viewpoint in a research lab-

oratory, the team has produced a working new process in Stage 1. Now, the team has also

designed in the first loop of this stage a prototype production facility. Over the course

of Loop 2, the team operates the prototype production facility to ensure that the initial

research is correct and scalable. The reason for producing a full-scale production facil-

ity is the same between manufacturing and finance; it is to ensure our initial findings can

be turned into a profitable, working system. Loop 2 is also optimizing signals, including

what mix of indicators gives the highest performance, using a subset of the data and mul-

tiple objective functions:

● Returns, excess returns, and volatility.

● Portfolio tradability (average holding period, percentage of float and total volume,

number of trades, turnover, etc.).

● Sharpe and Sortino ratios.

● Stability of the system.

13.5. LOOP 3: Generate Simulated Track Record

STEP 1: Document Data Maps, Result Tables, and Hardware Requirements

STEP 2: Document Cleaning Algorithms

STEP 3: Perform Out-of-Sample Test

STEP 4: Perform Regression Test of OS Test Results against IS Results and SPC

Outputs and Shadow Trade

Loop 3 is the final out-of-sample test of the process. This is done to measure and vali-

date how the design process will actually perform in production. The outputs of this stage

should be considered proxies of the actual outputs of the finalized system. This loop is

critical to ensure that the previous loops have not produced a system that has been opti-

mized for a specific set of data or economic conditions that may or may not occur in the

future. This problem is called overfitting the data, which is very common in time series

analysis, neural networking, and optimization where algorithms routinely predict 95% of

a processes variation historically and 0% in the future, which is only discovered by per-

forming out-of-sample testing. Over each backtest, the product team should again create

individual run packs, computer folders with unique names for each run. Included in the

13.5. LOOP 3: GENERATE SIMULATED TRACK RECORD