136 CHAPTER ◆ 1 3 STAGE 2: Overview
run pack should be a copy of the prototype software and data used to generate the run,
and a run control document, containing:● The unique run number and date/time.
● A summary of run pack contents, signed off on by all team members.
● Reports, graphs, tables of trading system performance results (probably in Excel for
ease of statistical analysis).
● Details of key assumptions.
● Excel spreadsheet with trade selections for backtests and shadow trades.13.6. Outputs/ Deliverables
At the completion of Stage 2, we will encounter the second gate and, as a result, a gate
meeting. At this meeting management will expect several deliverables, including:● Full description of the database and real-time data feeds, including comparison of
alternative data sources.
● Full investment policy outlining the investable universe.
● Complete definition of the data cleaning rules and optimization methodologies. This
may be included in an updated version of a business rules catalog.
● A data dictionary, data and data flow maps, and a database design document.
● Documentation outlining the in-sample/out-of-sample tests with performance met-
rics, including mocked-up reports, SPC charts, and user interfaces.
● Full prototype, including proof of concept, trade selection charts for shadow trades,
including results of regression tests versus prototypes delivered at the completion of
Stage 1.
● Identification of software quality metrics.(For best practices regarding writing stage deliverables, refer to Chapter 8.)13.7. Summary
Backtesting is essentially the make or break stage of the trading/system design and devel-
opment project. Either the system generates acceptable performance or it does not. If the
system does not, then the project will be killed or sent back to Stage 1. This spiral also
contains the highest risk as testing migrates from a very small sample data set used to
prove components in Stage 1 to a real set of data and all of the issues that go along with
real data. The initial test data could be an anomaly versus real data that does not prove the
concept. Product teams often try to force the system to work with the real data, attempt-
ing to find anomalous data where it proves successful. But, the structure of our methodol-
ogy prevents endless testing and modification; product teams report progress and cannot
go back. The monitoring and formalized reporting of results of K| V is an important ben-
efit over most other software development methodologies. After this loop, the building of