158 CHAPTER ◆ 1 6 Perform In-Sample/Out-of-Sample Tests
only in sample, it may be allocated additional resources for continued research and/or
sent back to Stage 1. If, however, the system proves to be unprofitable both in sample as
well as out of sample, management will likely scrap the project altogether mainly due to
the nonscalability of the trading idea.
The purpose of backtesting is not simply to identify returns, standard deviations, and
drawdowns, but also to identify the common cause and any special cause variations in
the trading/investment system. Key components of performance measurement will be the
expected fees charged by brokers, exchanges, regulators, custodians, and auditors as well
as taxes. Backtesting should always be performed in light of these fees.
Modern computer modeling techniques have allowed very complex models with many
parameters to be estimated and fitted. Beyond this, the creation of new models itself has
been automated. The result is that model selection and validation has become more dif-
ficult. The concept of “ model error ” while once mostly an ignored issue must frequently
be directly addressed.16.1. STEP 3, LOOP 1: Define Testing Methodology
In the original testing conducted in Stage 1, the product team used a small set of clean data to
prove the calculations and trading algorithms. The focus now is on tests of the performance
of the entire trading/investment system using historical data. To perform this task, the team
should lay out a testing methodology. The question is the following: how do we take the
black box test from K|V 1.4 and roll it into a multiyear, multi-instrument test?
The first step is to build a reliable set of instruments to test the algorithm against. We will
call this set the investable universe, and it will be used for the entire backtesting process. (The
trading system will only buy and sell instruments within this universe. In Stage 4, risk manag-
ers will monitor the trading/investment system ’ s positions to ensure compliance.) The selec-
tion of the instruments should be based on the following items, that is, investable instruments:● Must have data with both a beginning price and an ending price over the appropriate
time interval. Which is to say, we will select only those instruments that have suf-
ficient price/valuation data to calculate returns.
● Must pass a liquidity screen. A screen may filter out low priced stocks, bonds with
limited issuance, or contracts with low outstanding notional value. For instruments
that are inherently illiquid, additional slippage must be added to execution prices.
A liquidity screen should make sure that as the pool of investment capital grows, the
system will scale properly. As a general rule, the trading/investment system, in its
maturity stage, should not purchase more than 2% of the outstanding shares or open
interest (except for market making systems).
● Must have the data necessary to calculate the factors or signals that comprise the
trade selection algorithms.The team can either build this data set themselves, or they may be able to purchase ven-
dor software to run these screens. Instruments that pass these filters become the invest-
able universe, a locked-down data set which the team will use for all backtesting and
regression testing from this step forward.
The next step is for the instruments in the investable universe to calculate the return,
volatility, and drawdowns for each instrument for each time period, adding in dividends.
All this must be done before backtesting begins. For stocks, this is trivial. For derivative
products that include hedging these calculations can be very complex.