215
All approved designs and software should be placed under configuration management,
which is accurate with respect to the numbering or naming protocols of executable pro-
grams, software modules, software units, and associated documents. Software development
libraries must provide for proper handling of code and documentation from their initial
approval until they have been incorporated into the final version.
QA should also include auditing of the test results of individual units, modules, and
components, and integration of modules and the system as a whole. QA assures that:● Test procedures truly follow test plans and that these procedures are verifiable.
● Tests are conducted on the correct version of software.
● Changes to code are made properly, and that no unauthorized changes are made.
● Nonconformities discovered during testing are recorded and that procedures to address
nonconformities are followed.
● Regression testing is conducted to assure nonconformities have been corrected.
(This is to say that resolution of all nonconformities must take place prior to Step 4.
QA is not debugging.)
● Test reports are correct and complete.Lastly, QA will include regression testing of the outputs of the trading/investment system
against gold standard, black box results experienced in Stage 2.
Tools exist to assist the QA process and the product team should evaluate its need for
proprietary tools versus those available off the shelf. Useful tools include QA audit and
inspection checklists as well as automated code screens, reports, and calculation testers.23.2. LOOP 1: Test against Black Box
Results from Stage 2
Over the course of Loop 1, the development team has built and tested the business layer of
the trading/investment system. To ensure that this layer correctly implements the quantitative
methods and business logic designed over the course of Stage 1 and 2, the financial engineers
perform regression testing using static data. Validation of the business layer will demonstrate
the ability of the system to achieve the planned results, that is, to do what it is supposed to do.
QA regression tests take known data (i.e., data used during K | V 2.4) and known
model inputs for calculations. The financial engineer leads QA regression testing and
makes sure the tests are done to specification, mapping inputs to outputs. For example,
the financial engineers may run 100 products and five years of data through the production
software to verify that all inputs lead to all the correct outputs. If the outputs are not iden-
tical, the team will have to hunt down the differences. A normal source of difference is in
rounding algorithms. Rounding algorithms in C of COTS components may not match
the rounding algorithms in Excel or MATLAB. All differences in rounding algorithms,
interpolation algorithms, and precision tolerances in optimization should be investigated
with a documented conclusion. The financial engineers should keep a list of known dif-
ferences and their causes for future discussion with the product team or top management.
Depending on those causes, they may or may not need to run additional regression tests,
say for 1002170 instruments and ten years.
A QA test of the business layer should regress the mean returns, standard deviations
of returns, Sharpe ratios, etc., and range of the time series of outputs using SPC. Using23.2. LOOP 1: TEST AGAINST BLACK BOX RESULTS