166 CHAPTER 5
Meta-analysis
A research method that statistically combines
the results of a number of studies that
address the same question to determine the
overall effect.
can modify a patient’s behavior. However, like case studies, single-participant
experiments consider only one individual, so the results can be specifi c to that
individual and based on neurological, psychological, or social factors that may
not apply to others, or at least not in the same combination. Moreover, other
confounding variables—such as the therapist’s enthusiasm for the treatment,
rather than the treatment itself—limit the generalizations that can be made.
When the results of single-participant experiments are published, it is often with
the goal of informing clinicians about possible interventions that might work
for patients with the same problem and in similar circumstances.Meta-Analysis
Despite the best efforts of researchers to minimize confounds, the results of any
one study must be taken with a grain of salt—it’s not clear whether researchers
would obtain similar results if the study were undertaken in somewhat different
circumstances. Perhaps the signifi cant results found in one study were a fl uke—or
perhaps the lack of signifi cant results in another study was a fl uke. Such fl ukes
can arise, in part, because of the particular people who were studied. Just as
people vary in height and weight, they also vary in their behavioral tendencies,
cognitive abilities, personality characteristics, and symptoms. This variety means
that different samples from the same population can vary substantially, and when
a sample is relatively small, chance variation can produce an appearance of a
difference when such a difference doesn’t really exist, or can obscure the mea-
surement of an actual difference in the population. Moreover, sometimes an indi-
vidual study does not produce a signifi cant fi nding, but instead reports a near miss
(for example,p < .07, which is above the critical .05 cutoff). If enough studies
report similar near misses, there may be an effect, but it just isn’t being measured
well enough.
But how can researchers statistically evaluate more than one study that ex-
amines the same question? Meta-analysis is a research method that statistically
combines the results of a number of studies that address the same question. This
strategy can be especially valuable when some studies show an effect but others
do not (Rosenthal, 1991). Because a meta-analysis increases the size of the overall
data set, it can help to determine whether or not certain variables are related. In
many cases, when studies are considered together in a single meta-analysis, the
effect emerges loud and clear. A meta-analysis can uncover a relationship that
is not apparent in any single study, which considers only a single sample from a
particular population.
For instance, some clinicians have claimed that people with schizophrenia who
have more insight about their disorder and its consequences also display fewer
symptoms than do people with schizophrenia who have less insight. However, stud-
ies that address this claim have yielded inconsistent results: Some suggest a relation-
ship between insight and number of symptoms, whereas others do not. Researchers
conducted a meta-analysis of 40 studies on this relationship, and the results indi-
cate that greater insight is in fact associated with fewer symptoms overall (Mintz,
Dobson, & Romney, 2003).
However, meta-analyses have certain drawbacks. For one, there’s the so-called
fi le drawer problem. The results of studies that failed to fi nd effects may linger in
a researcher’s fi le drawer, never making it to publication and thus never being in-
cluded in a meta-analysis—and it can be diffi cult to estimate how many such nega-
tive fi ndings there are. For another, not all studies are conducted equally well (e.g.,
some have more confounds than others). Some researchers have therefore asserted
“garbage in, garbage out”: If the individual studies are no good, why should we be-
lieve that aggregating them is a good idea? Thus, although meta-analyses can reveal
underlying regularities in results from different studies, they must be followed up
with new studies that directly test their conclusions.
Table 5.1 provides a summary of the different research methods we’ve discussed.