1302 Testing Econometric Software
likelihood problems and giving them not to the package’s nonlinear least squares
solver, but instead to its maximum likelihood solver, would produce a method for
testing the efficacy of maximum likelihood solvers. To date, there has been no such
systematic testing of the maximum likelihood solver of any econometrics package.
Preliminary examination of this topic by the author suggests that many packages
have maximum likelihood solvers that tend to stop at points that are not solutions
but nonetheless declare that a solution has been found. The implications of this
line of research for applied econometrics would be enormous.
28.4.2 Random number generators
Random numbers form the basis of many econometric estimators, e.g., simula-
tion and bootstrapping. Yet the efficacy of these methods depends on the random
numbers being truly random. In fact, the random number generator (RNG) in an
econometrics package does not produce random numbers, but deterministic num-
bers. However, they are nonlinearly deterministic and can give the appearance of
being truly random. For a short primer see McCullough (2001) or McCullough and
Vinod (1999, sec. 5). The physics literature is filled with stories of bad simulation
results due to bad RNGs (see, e.g., Coddington, 1994, 1996). The economics litera-
ture has no such examples because there is practically no replication in economics.
When was the last time one economist took another economist’s simulation code
and swapped out one RNG for another RNG to see if the results would change?
Nonetheless, the literature on testing econometric software has found bad RNGs,
and the usual response by a developer is to put in a good RNG.
The primary method for determining whether an RNG produces “random” num-
bers is to take the numbers and apply a statistical test of some sort. As a simple
example, take 1,000 random numbers, compute the first-order autocorrelation
coefficient, and test the null that the coefficient equals zero. Rejecting the null
constitutes evidence that the numbers are not random; they are correlated. There
are many, many types of tests because there are many, many ways for a sequence
of numbers not to be random. One of the first standard collections of such tests
was given by the eminent computer scientist Donald Knuth in the 1981 first edi-
tion of his classic text (Knuth, 1998). Coding these many tests and applying them
was time-consuming and tedious. The random-number specialist George Marsaglia
(1996) collected several tests and coded them as “DIEHARD: a battery of tests of
randomness.” DIEHARD has been used to uncover bad RNGs in many statistical
and econometric software packages.
As the scale of computing has grown since then, the DIEHARD tests are no longer
up to the task of validating RNGs for the huge simulations that are run today. The
RNG testing program TESTU01 has been the new standard since its first release in
- The program is reviewed by McCullough (2006) and described in more detail
in L’Ecuyer and Simard (2007). Using TESTU01 requires a passing knowledge of C,
and is available freely. It comes with three batteries of tests pre-programmed, Small
Crush (which can take a few minutes), Crush (which can take a few hours), and
Big Crush (which can take a day or more). By contrast, on a modern PC, DIEHARD
takes several seconds. If an RNG passes Small Crush, then give it to Crush. If it