161
production-level code. From the manufacturing analogy, this is where the team builds the
prototype production facility. To summarize the standard steps for this loop, we recom-
mend that you:
● First calculate the returns and store them in separate table.
● Then, calculate all the factors or signals.In-sample testing will expose irregularities in the data (e.g., fiscal year changes), so that
they can make modifications. If the modifications require new quantitative methods, how-
ever, the project will need to revert back to Stage 1, and be reresearched and reprototyped
before moving again to the backtesting stage. For this step, we expect the appropriate
exception handling to take place, such as catching changes in reporting cycles. These
types of quality control issues are similar to manufacturing issues encountered when
going from a test beaker of chemicals to a 50,000 gallon batch.
After calculating the returns and factors, graph compounded performance by factor
or signal. For trigger trading systems, graph the linked trigger P & L versus the P & L of
the investable universe and graph the excess return versus investable universe. For filter
systems, graph the performance of the selected basket versus both the basket of unchosen
instruments and the investable universe. For multifactor systems, graph each quantile ver-
sus the investable universe. We recommend keeping it ordinal; the bottom quantile should
be decreasing, the top quantile should be increasing, to form a fan shape. If the graph is
not ordinal, then there is a problem with the indicator.
The team should perform this graphical statistical analysis for each indicator by itself
and with all indicators combined together. The team can use a design of experiments (DoE)
approach or fishbone diagram for individual (i.e., unblended) factors to investigate the inter-
action between the predictive capability of a given factor with other factors, such as eco-
nomic cycle, that have not been modeled in the system due to the lack of this data in real
time. Given a fishbone diagram, the team can decide which combinations of factors to look
at together. Also, because the trading/investment algorithms are locked after Gate 1, stick
with algorithms defined in K|V 1.2. At this point the team is optimizing the signals. The fac-
tor weightings may change, but not the underlying signal algorithm.
The team should decompose the returns, producing return charts and tables, to allow
the team to engage in introductory risk control discussions, such as is the algorithm over-
weighting technology stocks? To facilitate this, these returns should be calculated per
sector or per bond class, etc. The team should look not at the end returns, but rather how
these returns were created. Ideally, returns are generated by picking better securities ver-
sus over- or underweighting classifications.
The team is trying to figure if the algorithm has a bias. For example, in the basket,
what percentage is large cap, or small cap? In a basket of securities, what are the credit
ratings of the companies purchased—A, BBB, C? Is it effectively going long large caps
and short small caps? Is the algorithm effectively going long companies with junk credit
ratings and short companies with A or better credit? We recommend getting the positions
taken in statistical format and analyzing the outputs. There may be some unexpected, hid-
den factor model. Given the returns of the algorithms, there may be some principal com-
ponent analysis that explains most of the returns outside of the chosen factors or signals.
Ideally, graph the returns by as many classification factors as possible. This means
probably producing 30 to 40 graphs in order to determine how stable the trading algo-
rithm is, given different slices of the universe. From a database view, given a selected
basket, take from the table of everything you bought, SELECT WHERE Sector ‘ Oil ’.
SELECT WHERE Cap 1000000.
16.2. S T E P 3 , LOOP 2: PERFORM IN-SAMPLE TEST