ferent times. The characteristics that are shared by any one batch of
proposals vary and may make different criteria of evaluation more
salient. As a historian of China points out:
It does sometimes happen that we get some that are very close to
each other, and I always go back again and look at the ones that I
thought were really the best and really the worst and see if they’re
really all that much different. It’s like working yourself through
any batch of applications or papers or whatever: your standards
kind of evolve as you go through it. I don’t sort mechanically...
Until I’ve read the whole batch, I don’t even know exactly what
the standards are going to be.Maintaining consistency is also often at odds with the imperative
of cognitive contextualism discussed earlier, which requires that the
most appropriate disciplinary criteria be applied. A sociologist, re-
flecting on the challenge of consistency in criteria, observes:
We were taking different disciplines and trying to make the rules
up as we went along, really. We were saying, well, what counts as
ethnography in sociology isn’t what counts for ethnography in
anthropology. It was quite hard, really...toremain consistent
given that everybody had different consistencies, you know, we
were all trying to be consistent in our own ways.To respect “cognitive contextualization” means to adopt different
criteria of evaluation for different proposals. But the resulting in-
commensurability is very much at odds with a social science episte-
mology that would suggest that the same standards (about falsificat-
ion, for instance) should apply to all types of research.^35
The application of consistent criteria generated one of the three
serious conflicts that emerged on the panels I studied. A scholar de-
142 / Pragmatic Fairness