238 CHAPTER ◆ 2 6 Plan Performance and Risk Processes
All these tool sets, however, generate estimates and forecasts that are only snapshots in
time of what might be a poorly explained stochastic system (as determined by the r 2 of
the model). If an output of a trading/investment system has changed, the risk manager
must understand whether that change is within a normal range or due to special, or assign-
able, changes in the inputs. If inputs have changed, this may have caused the distribution of
the outputs to change. To make this determination, the risk manager looks not only at
returns and standard deviations, but also at many process outputs—sector weights, credit
score weights, average drawdowns, number of winning trades—and asks a simple ques-
tion: have the output distributions changed?
If the trading/investment system is not producing the same outputs again and again and
again, then the product team must figure out what is broken and how to fix it. A trading/
investment system may exhibit nonconformance, that is, be broken, if an input distribution
has changed, because, for example:● Other trading/investment systems have come online and are now competing.
● Competing systems have evolved to explain more variation. For example, competi-
tors may have changed from static to Bayesian statistics.
● Laws have changed. For example, Sarbanes-Oxley completely changed the meaning
of earnings estimate revisions.
● Exchange matching algorithms have changed and/or execution processes no longer
outperform their benchmark.
● Technology has changed from single to multiprocessor.Successful firms automate the process of monitoring and reporting portfolio statistics, trade
limits, and risk factors. Essentially, these reports will help risk managers understand whether
or not the system is working within specifications relative to the backtest, the index bench-
mark, and risk forecasts. These records prove conformance with expectations and document
nonconformance, including the types and nature of nonconformities, and actions taken to
repair or preclude its further use. There are three ways of dealing with nonconformity:● Finding and correcting the nonconformity (this requires at least a reversion to Stage 2).
● Authorizing its use, release, or acceptance if concession is obtained from a relevant
authority.
● Shutting down the system.A defect causing a nonconformity may cause the firm to rework or repair the trading/
investment system. The product team will have to demonstrate conformity of a previously
nonconforming trading/investment system that has been corrected. This must be done by
reverification through the same gates and acceptance criteria in previous stages.
Most trading system developers view that performance outputs of the system are for
the risk department and the sales department after the system is running; they believe
the machine is optimal and contains very little variation. The product team has built a
complete database on the performance of the trading/investment system and now must
continuously measure the ongoing performance of the machine versus the backtest data.
This continuous checking, this automated feedback mechanism, enables the people run-
ning the system to quickly identify problems, begin analysis, and take corrective actions.
This action may include a trader overriding the system. Risk and performance metrics are
not only for the risk department in well-run firms, but also by the traders that operate the