102 Metastatistics for the Non-Bayesian Regression Runner
The debate revolved around whether the evidence from that trial and the previous
history of non-randomized studies was “sufficient” or whether any other studies
involving randomization were necessary. The researchers were reluctant to con-
clude that the single trial and the previous studies using “historical controls” were
enough. Ware (1989), among others, observed that the randomization wasn’t
satisfactory and that one couldn’t rule out other explanations for the observed
outcomes. For instance, the sole infant not randomized to treatment was, coinci-
dentally, the most severely ill patient in the study. The implication was that, had
this one patient been randomized to ECMO, it is quite likely the child still wouldn’t
have survived.
Berry (1989), an advocate of Bayesian methods, harshly condemned the decision
to continue further study as unethical.^7
Most of the debate focused on the structure of the randomization, and revolved
around a very narrow “binary” question: “Did ECMO work?” or, possibly, “What
was the probability that ECMO works?” Both sides focused on whether the answer
was “yes” or “no.” The debate did not include, for example, a heated discussion
about the necessary prerequisites to be considered “eligible” for treatment. Even
if the researchers had used a more conventional randomization scheme, the study
would not have been able to provide a good answer tothatmuch more difficult
question.^8
3.1.2 The metastatistics literature
It is likely that much of the philosophical discussion on “induction” or “metastatis-
tics” is somewhat unfamiliar to regression runners – it was to me. Moreover, often
the metastatistics debate seems to involve few participants of the practical sort. As
a consequence, many of the case study examples debated by philosophers of induc-
tion or statistics are drawn from physics; I am sure this is true in large part because
physics has had some success – it is easier to debate “how to get the answer right”
in a science when a consensus exists that, at some point, someone got it right.
Such cases are rare (non-existent?) for low sciences like medicine and economics.
As part and parcel of this general tendency, the types of problems considered
in the metastatistics literature often seem far removed from the types of prob-
lems confronted by economists of my stripe – the “practically minded regression
runner.”^9
When (by accident) I began reading about the philosophy of statistics I was
surprised to discover:
- the vehemence of the debate, and
- the almost near-unanimous consensus that almost everything someone like
me – a “practically minded non-Bayesian regression runner” – understands
about statistics is wrong or profoundly misguided at best.
Concerning (2), consider the “stupid” inferences people like myself are “supposed
to draw” on account of not adopting a Bayesian point of view. One example is
inspired by an example from Berger and Wolpert (1988). Consider computing the