Palgrave Handbook of Econometrics: Applied Econometrics

126 Metastatistics for the Non-Bayesian Regression Runner

There are several points to make about these “experiments” from a non-Bayesian
perspective.

1. One point to emphasize is that in experiment A, the sample size is fixed. In
   experiment B, it waspossiblethat the same experimenter would have continued
   to draw balls from the urn if a third red ball had not been drawn.


2. In neither case is it correct to make a statement such as “Given the experimental
   results (of 9 black and 3 red) there is a 7.3% probability in experiment A (3.3%
   probability in experiment B) that the null hypothesis is true.” The hypothesis
   is presumably either true or false. The probability statements are statements
   about one particular “property” of a procedure. Whether it is a “good” procedure
   depends on a great deal more.


3. For many purposes, neither experiment is particularly “good.” It depends on
   the alternative hypothesis that is the salient rival, but it is easy to come up with
   cases where Type I and II errors are going to be rather large. Figure 3.4 displays
   the sampling distribution of the two estimators. Neither experiment is going to
   be good, for example, at detecting the difference between a true mean of 0.5
   and 0.51.
   Indeed, this was the non-Bayesian reaction to our earlier examination of
   ECMO: these experiments aren’t likely to settle a well-meaning debate. Some-
   times one is faced with a situation where one is trying to squeeze some inferential

0

.1

.2

0

.1

.2

0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1

Binomial experiment

Negative binomial experiment

Fraction observed in 100,000 trials

Fraction black

Graphs by type of experiment

Figure 3.4 The introductory puzzle – which data-generation process?