John DiNardo 129
What the results of the experiment would do to a non-Bayesian’sbeliefsis a
separate matter, but it is clear that he/she would find a “rejection” much more
informative in the first case.
By contrast, for the Bayesian who adheres to the Likelihood Principle, both exper-
iments provide thesameinformation: the posterior probability assigned to the
null hypothesis should be the same, regardless of which experiment is performed.
The Bayesian is free to ignore the “intentions” of the experimenter (that is, the
DGP), presumably “locked up” in the mind of the experimenter. Confronted with
the evidence consistent with the usual rejection of the null for the non-Bayesian,
the change in the posterior beliefs of the Bayesian would be the same under both
experiments.
An interesting debate on “optional stopping” can be found in the famous Savage
forum (1962, p. 70 ff.), where precisely this example is discussed. For Armitage, a
non-Bayesian, the DGP is very important and a flaw of Bayesian reasoning:
I think it is quite clear that likelihood ratios, and therefore posterior probabili-
ties, do not depend on a stopping rule. Professor Savage, Dr Cox and Mr Lindley
[participants in the forum] take this necessarily as a point in favour of the
use of Bayesian methods. My own feeling goes the other way. I feel that if a
man deliberately stopped an investigation when he had departed sufficiently
far from his particular hypothesis, then “Thou shalt be misled if thou dost not
know that.” If so, prior probability methods seem to appear in a less attractive
light than frequency methods, where one can take into account the method of
sampling. (Savageet al.,1962, p. 72)
G.A. Barnard, another forum participant – who originally proposed that the
two experiments should be the same (Barnard, 1947a, 1947b) and introduced the
notion to Savage – expressed the view that the appropriate mode of inference would
depend on whether the problem was really a matter of choosing among a finite set
of well-defined alternatives (in which case ignoring the DGP was appropriate) or
whether the alternatives could not be so clearly spelled out (in which case ignoring
the DGP was not appropriate.)^37
3.5.4 What probabilities aren’t – the non-Bayesian view
In a phrase, a Bayesian is more congenial to the notion that probabilities generated
in the course of hypothesis testing represent the “personal probability that some
claim is true or not,” while such probabilities are merely devices that help “guide
inductive behavior by assessing the usefulness of an experiment in revealing an
‘error.”’^38 One problem sometimes cited by Bayesians is that non-Bayesians don’t
understand what “probability” means. To put it succinctly, a “p-value” isnot:
- “The probability of the null hypothesis.
- The probability that you will make a Type I error if you reject the null hypothesis.
- The probability that the observed data occurred by chance.”(Goodman, 2004)
