136 Metastatistics for the Non-Bayesian Regression Runner
utilitiesU(d 1 )andU(d 2 )respectively. Then a randomized decision, takingd 1
with probabilityλandd 2 with probability 1−λwould have expected utility
λU(d 1 )+( 1 −λ)U(d 2 ). If the randomization is non-trivial, i.e. if 0<λ<1, then
randomization could be optimal only whenU(d 1 )=U(d 2 ), and even then a
non-randomized decision would be as good.Savage goes further: “It has been puzzling to understand, why, if random choices
can be advantageous insetting upan experiment, they cannot also be advantageous
in its analysis.”
There is much more to say: for instance, it may be useful to think about this class
of problems in terms of the severity concept that we introduced earlier. However,
it may be more instructive to consider two examples from real research. In one, I
identify the problem as lack of severe testing. In the second, I identify the problem
that the world is a “complicated place”: assertions that were felt to be well-grounded
by numerous studies seem less so in the face of a well-designed experiment.
3.6 Case study 1: “medication overuse headache”
In section 3.1.1 I briefly mentioned the case of ECMO – a treatment for infants
with persistent pulmonary hypertension whose success was initially uncertain,
but retrospectively seems of great benefit. Here I would like to consider a poten-
tially “mirror-image” case: a treatment is being administered that, in my view,
is potentially quite harmful. Also, I would argue, the literature is of unbeliev-
ably low quality. I locate the problem with the theory in the fact that, instead
of behaving like Mayo’s “error statistician” or engaging in “Peircean severe test-
ing,” the researchers began with a prior belief and then set about “updating” it.
It should be noted that none of the studies involving this topic used “Bayesian
statistics.”^40 Rather, the question is “Is there enough evidence to proceed with the
expert consensus or is more ‘severe’ testing necessary?”
This case is particularly useful because, as with many problems in medicine and
social sciences (and elsewhere), it involves a problem of dubious ontology (is there
“really” such a thing as medication overuse headache (MOH)?) as well as the prob-
lem of “new hypotheses” that “accommodate” the evidence instead of having a
theory held in advance that “predicted” the evidence (much like our “demon”
example in section 3.5.3).
A road map for what follows is:
- During a period of time when the field was considered a “backwater” a diagnosis
of MOH was developed. This theory argued that people with chronic severe
headache pain caused their pain by taking pain medication “too frequently”
and that, if they merely stopped taking the medication, their pain condition
would improve. - The evidence for this theory was that patients who agreed to stop their pain
medication had higher rates of improvement than those who didn’t. These