Panelists note that the evaluation process is not straightforward
because criteria change as one reads through the stack of applica-
tions. Most panelists take notes on each of the proposals and candi-
dates while reading and then periodically revisit their rankings in an
iterative process, with the result that, according to a political scien-
tist, “One really does come to the session with a fairly clear recall of
most of the applications.” As they discover what the pool of appli-
cants looks like, panelists often come to see that proposals shine un-
der very different lights, that different standards apply to different
proposals, and that proposals do not all win or lose for the same rea-
son. As one panelist puts it:
[One proposal] was very strong on originality, kind of ambitious,
and weak on the actual presentation. It was overwritten. And the
other proposal was actually uninteresting, but well presented.
They had opposite pluses and minuses, so the argument was why
should we fund this one and not do a similar kind of calculation
for the other one, but in the opposite way.Experience allows one to put incommensurable proposals like
these on the same matrix and draw conclusions (more on this in
Chapter 4).^29 Different characteristics—for instance, originality or
significance—can push a proposal above the proverbial line. The
evaluation process is not consistent or linear. As a historian says:
If we could get fifty good, empirically grounded, significant pro-
posals that had a reasonable comparative dimension to them, I
would certainly go for those...[inpractice]it’samatter of pick-
ing among twenty flawed proposals. [A proposal] that’s sort of in-
ternally sophisticated and empirically important, but lacks that
comparative or fashionable focus, is something that I would prob-42 / How Panels Work