EXPERIMENTAL RESEARCHExperimenters often undertake manipulation
checks to increase internal validity. A manipulation
checkis a process to verify theoretically salient
variables (e.g., independent, dependent, and inter-
vening variables in hypotheses). Its purpose is to
verify measurement validity (e.g., variables truly
measure the theoretical concepts) of whether the
conditions of the experiment had the intended
effects, or the degree of its experimental realism
(experimental realism is discussed later in this chap-
ter). We have manipulation checks to make certain
that the variables and conditions in our experiment
operate as we intended and help us rule out possible
threats to internal validity.
We check “manipulations” (our measures and
interventions in an experimental situation) with
pretests, pilot tests, and experimental debriefing.
We might create a pretest of certain experimental
conditions. For example, you have a confederate
act as if he or she is disabled and have preliminary
research participants observe the confederate. As acheck, you ask whether the participants believed the
confederate was truly disabled or just acting. In the
study on neighborhood preference (see Example
Box 3), the researchers showed videos of neigh-
borhoods to a small number of people before using
the videos in the study. This was done to verify that
people recognized the racial mix and neighbor-
hood’s social class as the researchers intended. If
you plan to provide participants with written or oral
instructions in an experiment, you might pretest
them with a few preliminary participants. You can
inquire about the clarity of the instructions and
whether the participants understood them as you
intended.
A “dry-run” or pilot test of the entire experi-
mental procedure can be a manipulation check. Dur-
ing and after the pilot test, you look for potential
flaws, mishaps, or misunderstandings. You ask
whether all parts of the experimental situation went
smoothly and had their intended effects on partici-
pants. You may check to see whether participants
paid attention and accepted the “cover story” if you
used deception.
Experimental debriefing after a pilot test or the
actual experiment can be a manipulation check. To
conduct an experimental debriefing (unlike ethical
debriefing that emphasizes removing a lie or de-
ception), you interview participants about details of
the experiment. You want to learn what they thought
was happening, whether they felt fully engaged and
took the situation seriously, and whether they felt
any confusion, anxiety, or discomfort. You may
discuss compensatory behavior and demand char-
acteristics or diffusion of treatment in such inter-
views. At times, experimenters drop a participant
from study data if they learn that the participant mis-
understood a critical aspect of the experiment, saw
through the cover story of deception, or modified
responses because of demand characteristics (also
see discussions on reactivity later in this chapter). For
example, an experimenter may drop data of a par-
ticipant who revealed that she or he did not accept
the deception cover study but believed the study was
about reactions toward disabled people (which it
was) and responded based on that belief (i.e., showed
demand characteristics) (see Example Box 6, Who
Helps a Co-Worker Who Is Disabled?).TABLE 3 Internal Validity and External
Validity Issues
INTERNAL VALIDITY EXTERNAL VALIDITYSelection bias Population generalization
History effect Naturalistic generalization
Testing effect Theoretical generalization
Maturation effect Mundane realism
Instrumentation Experimental realism
Experimental mortality Hawthorne effect
Statistical regression effect
Diffusion of treatment
Compensatory behavior
Experimenter expectancy
Demand characteristics
Placebo effectManipulation check A separate measure of
independent or dependent variables to verify their
measurement validity and/or experimental realism.