1296 Testing Econometric Software
package correctly reported that the problem has no solution. If you were to estimate
a probit model, which package would you prefer to use?
Do not think that in the present day this lack of accuracy affects only nonlinear
problems. McCullough (2004a) reported on econometrics packages that produced
correlation coefficients larger than unity. Further evidence of inaccuracy of statis-
tical and econometric software will be presented in subsequent sections. For the
present, what is of interest is this: “How can a user protect him/herself against
inaccurate econometric software?” The answer is, “Test the software.” The natural
follow-up question, “How?,” is the subject of this chapter.
In its purest form, to test a software package is to give it some input for which the
correct output is known. In the aforementioned case of the Longley benchmark,
the input is the dependent and independent variables, and the correct output is
the coefficients that he calculated by hand. For some problems it is too difficult
to work out the correct answer, and an alternative is to have two (or more) inde-
pendent programming efforts reach the same conclusion. This is the method used
by Drukker and Guan (2003) to produce a benchmark for a particular panel data
estimator. This chapter will concern itself with the former method – known inputs
and known outputs. There are three classes of tests: introductory, intermediate,
and advanced. We provide an overview of each class, including a description of
inaccuracies uncovered, with directions to further reading for those who wish to
read up on the subject. Section 28.2 briefly introduces a few necessary ideas about
the numerical limitations of computational software. Section 28.3 discusses a set
of introductory tests, known as “Wilkinson tests.” Section 28.4 discusses a set of
intermediate tests. Section 28.5 discusses a pair of advanced tests, one for FIML
estimation and one for GARCH estimation. To illuminate the numerical issues
involved, Section 28.6 creates a benchmark for ARMA estimation. Section 28.7
offers some conclusions.
In what follows we typically do not mention specific software packages, even
when they were identified in the article or review that is cited. The reason for this
is that many of the errors we recount have long since been corrected. A notable
exception is Microsoft Excel, because Microsoft has a track record of not fixing
errors in Excel. Should Excel even be mentioned in this chapter? Yes. One need
only surf the web to find that many professors teach introductory econometrics
with Excel.
28.2 Computer arithmetic
Generally, computers do not store numbers that are perfectly accurate, nor are
the calculations performed on those numbers carried out with perfect accuracy. A
computer stores numbers in bits (a bit is simply a single binary digit that holds
either a zero or a one) and bytes (eight bits to a byte). Currently, most desktop
computers have a 32-bit word length, but there do exist 64-bit computers, and
soon there will be 256-bit computers.
To make ideas concrete, suppose that we have a 4-bit word, with each bit being
a zero or a one. All counting must be in base-2 with up to four places. Zero is