1316 Testing Econometric Software
Table 28.10 Packages that backcast
Package μφ θ
Benchmark 17.065503687 0.91492053108 0.58262567074
Package U 17.11392 0.917600 0.607982
Package V 17.08580 0.919264 0.595042
Package W 17.06372 0.884730 0.530030
28.6.5 Final thoughts on ARMA benchmarking
We have used analytic derivatives and extended precision calculation in Math-
ematica to produce much-needed benchmarks for ARMA least squares models.
Generally, packages can hit the point estimates for the CLS benchmark, but the
method of standard error calculation for most packages is unknown. Packages offer
disparate answers for the ULS benchmark because each uses its own specialized,
undocumented algorithm for backcasting.
While this is a damning indictment of customary practice in econometric and
statistical software practice, what can we hope for the future? Should we expect
developers to fix these problems? Generally, no. To rewrite the CLS and ULS code
would be very time-consuming and of little benefit. CLS is an approximation to ULS
which, in turn, is an approximation to maximum likelihood. Everybody should
be using exact maximum likelihood instead of CLS or ULS (Choudhury, Hubata
and St. Louis, 1999). There is currently no exact maximum likelihood benchmark.
Someone should develop it, and all packages should converge on it. A package
that does not offer exact maximum likelihood should either implement it, or bet-
ter document its existing CLS/ULS code, and ensure that it hits the benchmark
presented in this section.
28.7 Conclusions
We have seen that it is not safe to assume that econometric software is accu-
rate, and we have reviewed methods of testing econometric software, of which
there are far too few. While this chapter has primarily concerned itself with the
“known inputs – known outputs” approach to testing, it was mentioned that there
is another approach: two independently developed methods producing the same
answer. The former approach is very time-consuming and requires some knowledge
of numerical methods. The latter approach simply requires two (or more) soft-
ware packages, and the ability to use them correctly. This latter method has not
been much employed simply because of the dearth of replication in economics.
However, it is reasonable to expect that there will be much more replication in
economics in the future, and the relation between the accuracy of econometric
software and replication merits exposition here in the concluding section.
Over 20 years ago, Dewald, Thursby and Anderson (1986) attempted to replicate
many articles from theJournal of Money, Credit and Banking. Dewald, Thursby and
Anderson advised against the adoption of an honor system, whereby publishing