COMPUTATIONAL MODELING AND SIMULATION AS ENABLERS FOR BIOLOGICAL DISCOVERY 131
5.3.4 Model Comparison and Evaluation,
Models are ultimately judged by their ability to make predictions. Qualitative models predict trends
or types of dynamics that can occur, as well as thresholds and bifurcations that delineate one type of
behavior from another. Quantitative models predict values that can be compared to actual experimental
data. Therefore, the selection of experiments to be performed can be determined, at least in part, by their
usefulness in constraining a model or selecting one model from a set of competing models.
The first step in model evaluation is to replicate and test a computational model of biological
systems that has been published. However, most papers contain typographical errors and do not pro-
vide a complete specification of the biological properties that were represented in the model. One
should be able to extract the specification from the model’s source code, but for a whole host of reasons
it is not always possible to obtain the actual files that were used for the published work.
In the neuroscience field, ModelDB (http://senselab.med.yale.edu/senselab/modeldb/) is being
developed to answer the need for a database of published models used in neuroscience research.^32 It is
part of the SenseLab project (http://senselab.med.yale.edu/), which is supported through the Human
Brain Project by the National Institute of Mental Health (NIMH), the National Institute of Neurologist
disorders and Stroke (NINDS), and the National Cancer Institute (NCI).
ModelDB is a curated database that is designed for convenient entry, search, and retrieval of models
written for any programming language or simulation environment. As of December 10, 2004, it con-
tained 141 downloadable models. Most of these are for NEURON, but 40 of them are for MATLAB,
GENESIS, SNNAP, or XPP, and there are also some models in C/C++ and FORTRAN. Database entries
are linked to the published literature so that users can more easily determine the “scientific context” of
any given model.
Although ModelDB is still in a developmental or research stage, it has already begun to have a positive
effect on computational modeling in neuroscience. Database logs indicate that it is seeing heavy usage, and
from personal communications the committee has learned that even experienced programmers who write
their own code in C/C++ are regularly examining models written for NEURON and other domain-specific
simulators, in order to determine key parameter values and other important details. Recently published
papers are beginning appear that cite ModelDB and the models it contains as sources of code, equations, or
parameters. Furthermore, a leading journal has adopted a policy that requires authors to make their source
code available as a condition of publication and encourages them to use ModelDB for this purpose.
As for model comparison, it is not possible to ascertain in isolation whether a given model is correct
since contradictory data may become available later, and indeed even “incorrect” models may make
correct predictions. Suitably complex models can be made to fit to any dataset, and one must guard
against “overfitting” a model. Thus, the predictions of a model must be viewed in the context of the
number of degrees of freedom of the model, and one measure that one model is better than another is a
judgment about which model best explains experimental data with the least model complexity. In some
cases, measures of the statistical significance of a model can be computed using a likelihood distribution
over predicated state variables taking into account the number of degrees of freedom present in the model.
At the same time, lessons learned over many centuries of scientific investigation regarding the use of
Occam’s Razor may have limited applicability in this context. Because biological phenomena are the result
of an evolutionary process that simply uses what is available, many biological phenomena are simply
cobbled together and in no sense can be regarded as the “simplest” way to accomplish something.
As noted in Footnote 28, there is a tension between the need to capture details faithfully in a model
and the desire to simplify those details so as to arrive at a representation that can be analyzed, understood
fully, and converted into scientific “knowledge.” There are numerous ways of reducing models that are
well known in applied mathematics communities. These include dimensional analysis and multiple time-
scale analysis (i.e., dissecting a system into parts that evolve rapidly versus those that change on a slower
(^32) M.L. Hines, T. Morse, M. Migliore, N.T. Carnevale, and G.M. Shepherd, “ModelDB: A Database to Support Computational
Neuroscience,” Journal of Computational Neuroscience 17(1):7-11, 2004; B.J. Richmond, “Editorial Commentary,” Journal of Computa-
tional Neuroscience 17(1):5, 2004.