EVALUATIVE CRITERIA AND EPISTEMIC COMMUNITIES 8989CHAPTER 5
JUDGING QUALITY
Evaluative Criteria and Epistemic Communities
PEREGRINE SCHWARTZ-SHEA
In graduate school, I decided to pursue the area of “research methods” because I thought it
would help me to do better research. In my experience, having to teach a topic forces me to clarify
my own understanding. I also thought that enthusiasm about teaching research methods would
assist me in obtaining a job; and, indeed, I did end up teaching “Quantitative Methods” in both
of the positions for which I was hired. This decision to teach methods proved pivotal to my intel-
lectual journey in ways that I would never have anticipated in 1982.
Over the course of many years teaching research methods, I changed texts several times for
various reasons, but the language to which students were introduced remained constant:
operationalization of independent and dependent variables; hypothesis construction and tests of
statistical significance; the grounds for causal inference (time order, association, and non-spuri-
ousness); and criteria for evaluating empirical research—in other words, the reliability and va-
lidity of operationalized concepts and the internal validity and generalizability of results. Clearly,
this was the way to do “empirical” research in the social sciences. I was completely puzzled,
then, as I became interested in feminist research (which insisted on the observer’s standpoint, a
direct challenge to universality and objectivity) and philosophy of social science and experienced
the disconnect between what I had been teaching and what I was hearing in conference panels on
these topics. Specifically, if positivism was so discredited, why was this not reflected in the many
research methods texts I had perused and used?
My training in rational choice theory had equipped me with the faith that abstract theory
could be married to empirical research to produce objective, grand theory—the physics of the
social sciences. Reading interpretive work and feminist philosophy of social science shook that
faith. I first gave up on the possibility of a grand theory applicable to all historical epics and
cultures, then on rigid notions of objectivity. But still, research “standards” were the rope that
kept me from falling into the abyss of “postmodern relativism.” Reading interpretive research
engendered mixed emotions—an excitement at the possibility of doing research in new ways, and
fear: “How do I evaluate such work? I thought I knew what knowledge was and, instead, the
meaning of knowledge is contested!” It was as if the rules of academia had shifted underneath
me, and in my disorientation I grasped at “standards” as a way of understanding my identity as
an academic.
The language learned in quantitative methods courses not only disciplines researchers but
imprisons them. The desire to better answer such questions as “On what grounds does one distin-
guish ‘good’ from ‘bad’ interpretive research?” is what brought me to write this chapter.