of the experiment. There results a set of data in which the individual effects of sheep
on vegetation cannot be disentangled from the effects of rabbits, the total justifica-
tion of the experiment in the first place.
We have seen many such incomplete experiments set up, often at some expense.
They provide estimates of the effect on the vegetation of rabbits alone and of sheep
and rabbits together, but not of sheep alone. The effect of sheep alone cannot be
obtained indirectly by subtraction because that works only where the two effects are
additive (i.e. no interaction). But a significant interaction can quite safely be assumed
because each blade of grass eaten by a sheep is no longer available to a rabbit, and
vice versa.Necessary compromises
There are a number of problems that involve passive observation of a pattern or
process not under the researcher’s manipulative control. In these circumstances a
tight experimental design is sometimes not possible, or alternatively the problem may
not be open to classical scientific method. In many fields, for example astronomy,
geology, and economics, such problems are the rule rather than the exception. A
common example from ecology is the environmental impact assessment (EIA). As
Eberhardt and Thomas (1991) put it: “the basic problem in impact studies is that
evaluation of the environmental impact of a single installation of, say, a nuclear power
station on a river, cannot very well be formulated in the context of the classical
agricultural experimental design, since there is only one ‘treatment’ – the particular
power-generating station.” In fact the problem is even more intractable: EIA studies
do not test hypotheses. However, EIAs are still necessary. That they generate only
weak inference is no good argument against doing them.
Weak inference results also from a second class of problems: where tight experi-
mental design is theoretically possible but not practical. In such circumstances we
may have an unbalanced design, or poor interspersion of treatments, or insufficient
replication or even no replication. Again the results are not useless but they must be
treated for what they are: possibilities, which may be confirmed by further research.
This brings us into the realm of meta-analysis where replication of complete studies
is the answer ( Johnson 2002).
Weak inference is seldom harmful and can be very useful so long as its unreli-
ability is recognized. Weak inference mistaken for strong inference can be ruinously
dangerous.There are several possible analyses available for any given experimental or survey
design. Sometimes they give much the same answer and sometimes different
answers. The former reflect only that there is more than one way of doing things;
the latter reflect differing assumptions underlying the analyses. Hence, it is impor-
tant to know what a particular analysis can and cannot do lest one chooses the wrong
one. For example, chi-square tests are used only on frequencies (i.e. counts that come
as whole numbers); analysis of variance (called also A of V or ANOVA) can deal both
with frequencies and with continuous measurements. The Student’s t-test is a special
case of ANOVAand shares its underlying assumptions.
We will use ANOVAto introduce a broad class of analyses appropriate for the
majority of experimental and survey designs. Any statistical textbook will take this
discussion further and present additional analytical options.EXPERIMENTAL MANAGEMENT 279
16.6 Some standard analyses
WECC16 18/08/2005 14:47 Page 279