each coder, and subject each transcript to coding by one coder and
checking by another. The coding scheme took the following form:
each descriptive word used by a respondent was classified within
various categories and criteria of evaluation, many of which had
been identified from the observations and interviews using analytic
matrices. The codes were also derived inductively, with each coder
initially coding the same two transcripts and developing their own
coding scheme. The coding scheme was then standardized and the
transcripts split randomly between coders. After this initial round of
coding, the coders exchanged transcripts and verified and improved
each other’s work. For greater reliability, a third coder who was not
involved in the development of the coding key recoded all the tran-
scripts. Any conflicts were discussed with me and resolved. The cod-
ing key is available upon request.
Note that the interviews I conducted with program officers are
not included in our analysis of the frequency of use of formal and in-
formal criteria of evaluation and of epistemological styles. The five
interviews I conducted with three different panel chairs are included,
since they also served as peer reviewers and were asked about their
criteria of evaluation.
The last phase of data analysis was to connect the thematic
analysis of the transcripts with an analysis of the frequencies of
themes and criteria produced with the assistance of Atlas.ti. Com-
bining these two methods allows for depth in interpretation and
for systematic quantitative analysis of themes and patterns within
the data.
Limitations of the Study
In all phases of this study, I have been very aware of the many ways
in which I am indebted to work of researchers who have come be-
fore, even while I am critical of some of their blind spots. This
256 / Appendix