168 CHAPTER ◆ 1 7 Check Performance and Shadow Trade
and its outputs. (The concepts of control charts and C pk calculations will be covered in
the Stage 4 chapters. Many of the quality principles described in Stage 4 apply to Stage 2
performance evaluation.)
After this step, shadow trading may begin. The purpose of shadow trading is not to
confirm the profitability of the system, but to confirm that all the pieces and all the data
cleaning algorithms can be done in a timely manner to enable trading in a mock system
just to put the time element into the testing algorithm.
As with K | V 1.4.4, we again acknowledge that traders may well request to trade the
prototype. However, we again strongly caution that trading be delayed until after the Gate 2
meeting where top management can approve use of the system.17.4. Summary
The product must determine what data to collect; it is then necessary to clean the data,
run in-sample and out-of-sample tests, and analyze the outputs of the tests in order to
validate the ability of the trade selection algorithms, position management, and risk man-
agement logic to satisfy requirements. Quality testing demands that product teams com-
pare regression test results against gold standard results to ensure the correctness of data
cleaning algorithms. These tests will also indicate whether the current environment is
favorable for launch of the system. At the Gate 2 meeting, management will expect to see
graphical proof of the stability of the system and its performance characteristics.17.4.1. Best Practices
● Benchmark the benchmark. The benchmark may not consist of illiquid, untradable
securities, so you need to create a new universe so you can calculate the difference
between the benchmark and the investable universe.
● Measure the process capability of the system using statistical process control.
● Use regression testing to ensure that the current environment is suitable for launch.
● Create a gold standard package that includes the locked-down data and the perfor-
mance results.
● Shadow trade to check the ability of the system to produce signals in a timely fashion
that can be traded.