EVALUATIVE CRITERIA AND EPISTEMIC COMMUNITIES 107a term used in the criteria literature or in research methods texts. The literature makes clear that
interpretive researchers document their research processes, but such documentation is for pur-
poses other than attaining “reliability” or “objectivity,” as those terms’ epistemological ground-
ings conflict with interpretive presuppositions about the contextual nature of meaning making
over time and the role of the researcher in the research process.
A variety of techniques are intended to enable interpretive researchers to respond to the third
question, “How does the reader know that you didn’t look only for confirmatory evidence?”^23
Collectively, these techniques can be represented by the concept “negative case analysis,” used
by Lincoln and Guba (1985). Because of its Popperian connotations of falsifiability and a priori
deductive theorizing, what needs to be emphasized about negative case analysis is that it relates
directly to the researcher’s own struggle to make meaning—to make sense of interactions ob-
served in the field, of patterns he or she is seeing in documents or interviews, and/or of possible
inconsistencies resulting from triangulation. In these usages, negative case analysis is a technique
designed to prevent a researcher from settling too quickly on a pattern, answer, or interpretation;
the researcher consciously searches for any evidence—that is, the “negative” or negating case—
that will force a reexamination of initial impressions, pet theories, or favored explanations. A
slew of related techniques can accomplish the same goal.^24 For example, Miles and Huberman
use several phrases to refer to this idea: “checking the meaning of outliers,” “using extreme
cases,” “following up surprises,” “ruling out spurious relations,” and “checking out rival expla-
nations” (1994, 262). H. Becker (1998, 192–94) talks about “deviant cases.” The member check-
ing suggested by Erlandson et al. (1993) and others is intended to accomplish much the same
thing. Lincoln and Guba (1985) recommend what they call “peer debriefing,” that is, having a
colleague critique one’s preliminary analysis. Brower, Abolafia, and Carr also list similar tech-
niques to meet their criterion of “criticality”: “weighing competing interpretations” and “recog-
nizing and examining competing views or voices” (2000, 391).
Agar’s (1986) methodological treatise on ethnographic field research offers a slightly differ-
ent perspective on researchers’ meaning-making processes. While in the field, researchers need
to record their initial “breakdowns” in understanding due to their stranger status. As they become
more familiar with a setting, what was once “strange” becomes familiar and taken for granted;
that is, a state of “coherence” is reached.^25 Coherence can be further tested by researchers’ seek-
ing out additional social settings in which to assess their newfound cultural understanding(s).
Documenting the transformation from breakdown to coherence, from incidents of cultural awk-
wardness to smooth navigation of the culture, provides the evidentiary basis for demonstrating
that the researcher did not look only for confirmatory evidence.^26
As Agar’s work attests, techniques for responding to the third question are quite specific to each
data analytic form and even substantive area, making generalizations about techniques and criteria
relative to data analysis much more difficult than for the data accessing and generating stage.^27 Part
of the difficulty is the sheer number of analytic methods, from semiotics to genealogy to ethno-
graphic semantics (see Table I.1, xx, in the Introduction), in contrast to the comparatively limited
number of ways in which data can be accessed (via observation, interviews, or documents).^28 As
an example of a specific test developed for historical analysis of documents, Brandwein (see chap-
ter 12, this volume) discusses how the concept of “anachronism” emerged in the Cambridge School
as a criterion for assessing researchers’ representations and analyses of documents from earlier
time periods. For interpretive scholars working in public law and science studies, anachronistic
interpretations are poor interpretations because they project contemporary ideas onto historical
actors, thereby misunderstanding those actors’ meaning making. The anachronism criterion can be
used by scholars as a yardstick both in their own research and in evaluating others’ research.