3.5. Model diagnostics
Good model diagnostics are essential for model building. Essentially one is
looking at whether properties of the model are supported by the data. There is a cycle of
proposing a model, diagnosing the model, modifying the proposed model, diagnosing
the modified model and so forth. For complex, non-linear, hierarchical statistical models,
model diagnostics are essential, but there are few of them, and there is a great need for
fundamental research in this area (Little 2006).
There are several generic diagnostic procedures that currently exist in our
statistical portfolio, including:
- Validation (splits the data into two parts, one for model fitting, and one for
comparison to the fitted model predictions); - Cross-validation (successively deletes a datum, or group of data, with replacement,
and carries out a validation exercise for each deleted component); - Information criteria such as AIC, DIC, BIC, and posterior predictive loss (used to
compare several models but could also be used for model diagnosis; simultaneously
accounts for model fit and model complexity); - Posterior predictive distribution (a Bayesian version of a classical significance-
testing approach to testing hypotheses; see Gelman et al, 1996).
Although the aforementioned procedures exist, they were generally developed for
relatively simple models and data sets. These methods must be expanded upon, or new
methods developed, to accommodate the types of complex models that we will
encounter in sustainability research. That is, many models may produce different types
of predictions, potentially at different spatial and/or temporal scales, especially if the
overarching model(s) represents the coupling of multiple sub-models (see Section 3.7).
And, the models will likely be coupled to multiple data sources, also potentially varying in
their temporal frequencies, spatial scales, and levels of uncertainty (see Section 3.3).
Thus, the above procedures will likely be inadequate for diagnosing the overall
“behavior” of such complex models and/or their components.
3.6 Model assessment
The area of model assessment includes model selection using methods like AIC
(Burnham and Anderson, 2002) as well combining models using methods like Bayesian
model averaging (Hoeting et al., 1999). While much work has been done in model
assessment, the construction of new models requires new methods for model
assessment. For example, in hierarchical models, questions have arisen about how to
quantify the number of parameters in a model (Spiegelhalter et al. 2003). While
methods to address this issue have been proposed, these approaches have been shown
to be misleading when there are missing data and nonlinear model components. As an
example, consider a population dynamic model to estimate the number of elk in Rocky
Mountain National Park. The goal of the park is to maintain a sustainable population as
the number of elk skyrocketed after their predators were eradicated. A Bayesian
hierarchical model to predict the number of elk over time might involve one of more
dynamic models for aspects of population growth. The dynamic components of the
model typically have very different biological interpretations and thus scientists are
interested in choosing between them. Existing model selection methods often have
difficulties differentiating between these highly nonlinear models. New methodology