comments and observations show that the act of evaluating is an
eminently interactional and emotional undertaking, rather than a
cognitive process corrupted by extra-cognitive factors. More spe-
cifically:
- The context of evaluation is crucial.^44 Proposals are evaluated
relationally, in comparison with other proposals. The universe
of comparison is not stable, nor are the criteria. - Panelists support a pragmatic understanding of fairness that is
at odds with the ideal of considering only the intrinsic qualities
of proposals. Because many of the judgments are relational and
conjectural, panelists have to think strategically about what
they can realistically accomplish. - The influence of interpersonal relationships, including
clientelism, cannot be totally eliminated from the evaluation
process, because scholarly expertise is superimposed onto the
social networks of those who produce knowledge, including, in
this case, both applicants and evaluators. - Idiosyncratic evaluations are almost unavoidable, yet they are
usually presented in universalistic terms and seamlessly folded
into formal criteria of evaluation. - Respecting the customary rules of deliberations is crucial for
accumulating credibility with co-panelists (a necessary condi-
tion for making one’s label “stick” to a proposal) and for facili-
tating deliberations.
It should be remembered that belief in the fairness of the process
is not shared throughout academia. It is likely that scholars invited to
serve on panels have strongly internalized the view that peer review
“works.” Moreover, it is reasonable to surmise that the more success-
ful the scholar, the more likely he or she is to accept and endorse the
sacred values of the group—and further, that panelists are generally
very successful, given that program officers consider prestige when
Pragmatic Fairness / 157