Quality Money Management : Process Engineering and Best Practices for Systematic Trading and Investment

162 CHAPTER ◆ 1 6 Perform In-Sample/Out-of-Sample Tests

This loop is very much a “ field of NASCAR ” loop. Statistics and mathematics are

your tools to modify and build trading/investment systems. However, there is a large

amount of trial and error. Nothing replaces seasoned researchers, traders, and program-

mers that have proven they have the ability to tune an algorithm since they have done it

before successfully. Use stats and mathematics to make it work properly. Compounded

performances (when stated) were calculated by taking a hypothetical starting equity

amount and calculating the total return for the period. Each subsequent period then uses

the resulting equity balance as its start to calculate that period ’ s total return.

75,000

70,000

65,000

60,000

55,000

50,000

45,000

40,000

35,000

30,000

25,000

20,000

15,000

5,000

10,000

Jan. ’01 Jan. ’02 Jan. ’03 Jan. ’04 Jan. ’05 Jan. ’06

650

Return $ Return %

600

550

500

450

400

350

300

250

200

150

100

50

0

 50

bt_sow_upgrades and revisions2 S&P 500

bt_sow_upgrades and revisions2 – $10,000 starting equity

FIGURE 16-2

16.3. STEP 3, LOOP 3: Perform Out-of-Sample Test

Out-of-sample testing is done to ensure everything is working properly, with no adjust-

ments, with almost real-world data and samples. During the out-of-sample testing, no

more modifications can be made. Out-of-sample testing is backward looking (whereas

shadow trading is forward looking).

Trading algorithms and quantitative models must be examined against both in-sample

and out-of-sample data before progressing to the implementation stage. It is of utmost

importance to save some of the historical data for out-of-sample testing.

Often a trading idea at inception belongs in group three, but arrogance leads us to

believe that our idea falls in group one. And so it can at times be tempting to manipu-

late historical data and find periods of time where a particular system shows profitability.

Manipulation of this sort will lead to incorrect conclusions and furthermore will certainly

generate inconsistent if not disastrous results. Bad ideas are not all bad. Knowing what

does not work can sometimes be a giant step forward. A complete failure may be the

exact catalyst toward finding a profitable solution.