212 Recent Developments in Density Forecasting
forecasts, again under quadratic loss (see Clements, 1997, for an extension), the
most common evaluation method (see Nordhaus, 1987) is to test whether revi-
sions to successive point forecasts of the same event are independent. Clements
and Hendry (1998, Ch. 3) provide a textbook discussion of these tests. Patton
and Timmermann (2007) establish tests of point forecast optimality when the loss
function is unknown.
In turn, theex postevaluation of rolling density forecasts has begun to attract
considerable attention, and there now exist established evaluation methods based
on both the probability integral transforms and the logarithmic score, as we
review below, although there remains some uncertainty about their implemen-
tation in practice and their relative merits (see Gneiting, Balabdaoui and Raftery,
2007).
The genesis of these evaluation tests, as indicated in Dieboldet al.(1998), was
the literature on the evaluation of interval forecasts and probability forecasts. Since
these tests can also be applied to density forecasts, as a density forecast can always
be reduced to an interval forecast, they also constitute a means of evaluating
density forecasts. We therefore start our review of extant evaluation methods, in
section 5.4.1, with interval forecasts. Interval evaluation tests also serve as the basis
for tests of probability event forecasts. But since there are an infinity of possible
interval forecasts implied by a given density forecast, rendering it impracticable to
test all but plausible (or at least a finite set of) intervals, we then move our attention
to “whole” density evaluation methods. In the ensuing discussion we distinguish
between distributional (unconditional) and dependence (conditional) aspects of
the evaluation tests (see Giacomini and White, 2004).
In contrast, little attention has been paid, at least explicitly, to the fixed-event
aspect of density forecasts. This is despite the availability of the aforementioned
tests for the efficiency of fixed-event point forecasts – the testable proposition
(for weak efficiency) is that, under quadratic loss, forecast revisions should be
uncorrelated with past forecast revisions.
Extending efficiency tests to the density case, say using the KLIC (discussed
again in more detail below) to measure revisions to successive densities (as in
Lahiri and Liu, 2006), is the subject of ongoing research and, to date, there are
no established tests to review. But it does appear that KLIC revisions to successive
densities do not convey any information on forecast efficiency since conditional
variance forecasts, unlike conditional mean forecasts, are predictable even when
the forecaster is assumed efficient (see Mitchell, 2007c). Consistent with the “fan”
shape of density forecasts published by the Bank of England and others, condi-
tional variance forecasts decline as we get closer to the event of interest. This
“trend” precludes testing the efficiency of density forecasts, as with point fore-
casts, simply by testing the independence of revisions. But Mitchell (2007c) does
note that fixed-event density forecasts can always be evaluated similarly to fixed-
event point forecasts by reducing them to an event forecast. As we briefly explain
in section 5.4.1, fixed-event probability event forecasts can be evaluated just like
fixed-event point forecasts. Variance rationality has been examined by Batchelor
and Zarkesh (2000).