116 CHAPTER ◆ 1 1 Check Performance
is important. Fast track projects are particularly vulnerable to shorting quality assurance,
even though poor quality is one of the most common reasons for finishing late.^1
Preventing or detecting and removing defects early allows product teams to realize
significant schedule benefits in later stages. Reworking defective trading/investment sys-
tem requirements and design can consume nearly 50% of the total cost of development.
As a rule of thumb, every hour spent on defect prevention in Stage 1 will reduce repair
time anywhere from three to ten hours later in Stage 3 or 4 of K|V.^2
Design walkthroughs are useful for rapid development because teams can use them
to detect defects before backtesting begins in Stage 3. The earliest time that testing can
detect a requirements defect is after the requirement has been specified, designed, and
prototyped, that is, a walkthrough can detect a requirements defect at a specification
time.^3 Furthermore, research shows that walkthroughs can find between 30 and 70% of
the errors in a software program. We expect that results would be similar for trading system
development.
Inspections are a kind of formal technical review that has proven to be extremely
effective in detecting defects throughout a project. During inspections, product team mem-
bers use checklists to systematically review trading/investment system design compo-
nents, each person identifying defects in logic, data, mathematics, prototypes, or design
as they go. Throughout the inspection, team members should gather data on the number
of defects, hours spent correcting those defects, and hours spent on the inspection proc-
ess, so that later management and teams can analyze and improve the velocity of trading
system design and development.^4 Again, ample amounts of research show that inspec-
tions find from 60 to 90% of defects and reduce schedules by 10 to 30%. Rather than
slowing things down, quality assurance speeds things up and speed is vitally important.
Detection of defects in trading system design early on, when they are easy to find and
fix, will relieve the need for time-consuming and expensive rework and redesign down
the road. Just as unit tests serve as compilable documentation for the internals of the sys-
tem, acceptance tests and performance runs serve as compilable documentation of the
performance of the trading/investment system.11.2. Spreadsheet Testing
As Dr. Ray Panko points out:consistent with research on human error rates in other cognitive tasks, laboratory
studies and field examinations of real-world spreadsheets have confirmed that
developers make uncorrected errors in 2–5% of all formulas ... Consequently,
nearly all large spreadsheets are wrong and in fact have multiple errors...
Spreadsheet error rates are very similar to those in traditional programming.
However, while programmers spend [25]–40% of their development time in testing,
testing among spreadsheet developers in industry is extremely rare... [but] the
only proven way to reduce errors dramatically is testing.^5To programmers, “ testing ” means what Panko calls execution testing. In execution test-
ing the tester tries several input values to see if they produce correct results. However,
how does the developer know when a result is correct? Developers refer to this as the oracle
problem. An oracle is a way of knowing when results are correct.
For spreadsheets, there seldom are obvious oracles. Runtime errors do not happen.
Rather, logic errors result in incorrect calculations. Team testers must compare results