36
room for other errors in the assessment process. This is why it is crucial to follow a
strict process when choosing or developing an assessment tool.
If we select a measurement tool that is not right for our purpose or if we design
a tool with bad questions, it will only make a messy situation worse. Mistakes are
easy to make when choosing and designing an assessment. If we make too many of
them, the numbers we collect will be useless.
This chapter is an introduction to the basics of developing a specific type of
assessment – ratings that use a multiple-choice format. These tools are by no means
the only way to assess skills, but they have distinct advantages. They can be effi-
ciently given to a large number of people. They are transparent and can easily be
shared with other researchers. And the results can be compared across different set-
tings and across separate studies.
The discussion in the chapter will alternate between two main types of multiple-
question assessments: self-reported questionnaires and third-party reports. Below
are examples of each from the surgical education literature:
Self-reported survey questionnaires:
Researchers at Stanford University examined whether having “grit” was an impor-
tant predictor of mental well-being for general surgery residents [ 13 ]. Grit –
defined as perseverance and a passion to pursue long-term goals – was measured
by giving each resident an eight-item questionnaire previously developed by psy-
chologist Angela Duckworth [ 6 ]. Each item on the questionnaire contained a
statement (e.g., “I am diligent” and “I finish whatever I begin”) with response
choices ranging from “not like me at all” to “very much like me.” Responses to
each item are scored from 1 to 5, using a simple rubric, and the average score
across all items provides a Grit score.
Later in residency, the study participants were given questionnaires on burnout and
psychological well-being. These questions followed a similar format. The
authors found that the association between grit and mental well-being was sig-
nificant. Residents who reported higher levels of grit reported lower levels of
burnout and higher levels of mental well-being.
Third-party reports:
Third-party reports can be used to collect measures on soft skills or personality
traits such as grit – observers rate subjects using questionnaires similar to those
used to collect self-reports. For example, teachers can be surveyed on individual
student attitudes and behaviors (e.g., [ 11 ]).
Third-party reports have another key use: performance ratings. For example,
researchers at Southern Illinois University have developed a series of operative
performance ratings that are available via the American Board of Surgery [ 10 ].
Residents (or any surgeon) can be monitored (either live or via video) during
surgery (real or simulated) and have their performance rated using a standard
scoring sheet that follows the steps of the surgery.
This chapter draws heavily on the literature surrounding the design of self-
reported measures. However, the principles discussed – regarding the selection and
design of an assessment – apply just as well to third-party reports.
C. Hitt