120 CHAPTER ◆ 1 1 Check Performance
box test on a software design the testers only know the inputs and what the expected out-
comes should be and not how the program arrives at those outputs. The testers do not ever
examine the programming code and do not need any further knowledge of the program
other than its specifications. A black box tester typically interacts with the prototype soft-
ware through a locked-down user interface, providing inputs and examining outputs for cor-
rectness. For consolidated trading system prototypes, black box testing will almost certainly
be the only appropriate type of testing. The advantages of this type of testing include:● The tests are unbiased because the tester ’ s perspective is independent from that of
the developer.
● The testers do not need knowledge of any specific programming languages.
● Test cases can be designed as soon as the specifications are complete.
● Moderately skilled testers with no knowledge of hardware, software, or quantitative
finance can perform tests.A disadvantage of black box testing is that many possible scenarios are unrealistic
because it would take an inordinate amount of time; therefore, many logical program
paths may go untested.
In the end, acceptance tests are black box tests that verify that the requirements set
forth in the Money Document and the evolving description and documentation of the trad-
ing/investment system are met. Acceptance tests are written by folks who do not know
the internal mechanism of the system, that is, not the quant and not the programmer and
most likely the trader and marketer or testers external to the team.
The final loop through this step double checks all of the calculations, verifying
that the consolidated prototype is working properly. Also, during this final pass we will
generate all of the charts and supporting documentation. The results of this acceptance,
black box test will form the baseline or gold standard, the trusted results of the trading/
investment system performance, against which backtesting results, in the next stage, will
be compared, using a regression test. If the results in backtesting differ from the gold
standard, an error condition will exist. How else would the team know if everything is
running right?11.6. Trading the Prototype
After acceptance, we acknowledge that, in theory, traders and/or seed capital providers
could demand to begin trading the system using the consolidated prototype. After all, the
highest priority is to satisfy the seed capital investors through early and continuous deliv-
ery of working trading system prototypes. However, we strongly caution that trading be
delayed at a bare minimum until after the Gate 1 meeting where top management can
approve its use. Nevertheless, the product team must communicate clearly that trading of
a prototype assumes no automation of order entry, no risk or algorithm monitoring with
SPC, and no portfolio management, so results will vary greatly.
The primary purpose of trading the prototype should be only to fully understand the
performance monitoring tools that will be required in the later stages. Be aware, though,
that the performance of the system in probationary trading at this point will not be indica-
tive of the performance of the completed system. Trading may occur for a limited period
of time and with a small amount of capital in order to more fully understand the behav-
ior of the system and as mentioned to understand what tools will be needed to manage