3

Introductory Remarks on Metastatistics

for the Practically Minded Non-Bayesian

Regression Runner

John DiNardo

Abstract
It would appear that much debate among practically minded researchers in economics, social
science, and in other fields, is rooted in (frequently) unstated assumptions about the underlying
philosophical justification for the statistical procedures being debated. In this chapter, I try to
provide a simple non-technical introduction to some long-standing debates about “metastatisti-
cal” questions, especially those that divide (some) “Bayesians” from (some) non-Bayesians while
attempting to draw out some implications for the “practically minded non-Bayesian regression
runner.” Some of the issues which have prompted the most raucous debate in philosophical cir-
cles include: the meaning of “probability,” the importance or unimportance of pre-designation
(pre-specified research design), the role of “models,” and the practical value of hypothesis testing
and other common statistical practices. I discuss some of the links between these philosophical
views and actual practice and consider two different case studies – one from medicine and another
from labor economics.

3.1 Introduction 99
3.1.1 Life, death, and statistical philosophy: an example 100
3.1.2 The metastatistics literature 102
3.2 Six surprising ideas and one puzzle 105
3.2.1 Six surprising ideas 105
3.2.2 An introductory puzzle 107
3.3 What is statistics good for? 108
3.3.1 What’s utility got to do with it? 110
3.3.2 What is statistics good for? A non-Bayesian view 112
3.4 A few points of agreement, then... 113
3.4.1 Kolmogorov’s axioms 113
3.4.2 Definitions of probability 115
3.4.3 Aleatory or frequency-type probabilities 115
3.4.4 Objective, subjective, or “it depends” 116
3.4.5 Epistemic probability 117
3.4.6 Conditional probability, Bayes’ rule, theorem, law? 118
3.4.7 Reasoning or estimating with Bayes’ rule? 120
3.5 The importance of the data-generation process 123
3.5.1 An idealized hypothesis test 123
3.5.2 The introductory puzzle revisited 124

98