150 Metastatistics for the Non-Bayesian Regression Runner
While not rejecting the usefulness of statistics outright, Keane argues:that determinations of the usefulness of ...“well-executed” structural models –
both as interpreters of existing data and vehicles for predicting the impact of
policy interventions or changes in the forcing variables – should rest primarily
on how well the model performs in validation exercises. By this I mean: (1) Does
the model do a reasonable job of fitting important dimensions of the historical
data on which it was fit? (2) Does the model do a reasonable job at out-of-sample
prediction – especially when used to predict the impact of policy changes that
alter the economic environment in some fundamental way from that which
generated the data used in estimation?
My use of the word “reasonable” here is unapologetically vague.I believe that
whether a model passes these criteria is an intrinsically subjective judgment, for which
formal statistical testing provides little guidance. This perspective is consistent with
how other sciences treat validation. (Keane, 2007, p. 32; emphasis added)Indeed, Keane provides an illuminating illustration of this view by discussing an
example of estimating the parameters of a life-cycle human capital investment
model. After describing how the simplest version of the model fails to fit the data,
he goes on to explain:
by adding a number of extra features that are not essential to the model, but
that seem reasonable (like costs of returning to school, age effects in tastes for
schooling, measurement error in wages, and so on), we were able to achieve
what we regard as an excellent fit to the key quantitative features of the data- although formal statistical tests still rejected the hypothesis that the model
is the “true” data generating process (DGP). Despite these problems, there is
nothing to indicate that the profession might be ready to drop the human capital
investment model as a framework for explaining school and work choices over
the life-cycle. (ibid.,p. 33)
As one might expect, Keane does not set forth a specific context in which one
might find estimates of such a model “useful,” or to what extent, if any, the infer-
ences drawn from such an approach should influence the choices we make or
what we advocate to others. Surely the model with or without amendments can’t
be “reasonable” forallcontexts.
The “metastatistical” question is “How much confidence should one have in a
judgment supported by such an approach?” The answer, to say the least, is not
obvious.
Notes
- The original author(s) of the quote are unknown. “The very model of the anonymous
aphorism” (Koenker, 2007). - The number of Bayesian discussions are too numerous to list; nearly every book by a
Bayesian has some discussion of metastatistics. A few books I found helpful: Berger and