37There is an important type of assessment that we will not explore in great detail:
standardized tests of knowledge and reasoning. The design principles we discuss
apply to tests as well, but standardized tests have a series of other properties that
make them considerably more complicated to create than surveys. That said, even if
your interest is in the kind of knowledge captured by a test such as the American
Board of Surgery In-Training Examination, the lessons of this chapter should prove
useful.
This chapter poses nine key questions. These questions follow a sequence. They
lay out a step-by-step process for choosing and designing an assessment. If you
commit to finding clear and convincing answers to each of these questions, you will
be following a rigorous process that is used by leading researchers from social psy-
chology to medical education.
This chapter is introductory and intentionally nontechnical. It will steer you to
more detailed guides and resources. As you search for answers to these nine key
questions, you’ll also discover additional resources on your own. You’ll find none
more helpful than the work of Hunter Gehlbach, Anthony Artino, and colleagues
(e.g., [ 1 , 2 , 9 ]). Their work heavily influences the structure of this chapter, and their
research again and again points to a common theme. Designing an assessment is a
social endeavor. It cannot be done alone. You will need to rely heavily on colleagues,
experts, and your target audience for feedback and advice. You may need to find a
statistician to work with, once data is collected. And you will rely on peers to review
your findings, should you attempt to publish research based on the data you
collect.
Nine Key Questions of Assessment
The process of designing assessments and collecting data can be long and compli-
cated. But each step comes down to very basic principles:
- Who do you want to assess?
Define your target audience. Where are they in the learning process; what is
their professional status; what are their demographics and educational back-
grounds, etc.? - What are you trying to measure?
Define the attitude or skill or competency that you want to assess. This is
called your “construct.” It is likely complex, which means it has many compo-
nents. Give it a clear name, and clearly spell out the components that make up
the construct. - How have other researchers measured what you’re trying to measure?
Conduct a thorough literature review. Identify tools that others have used,
and draw upon previous research to improve how you define your construct. - What does your target audience think about what you’re trying to measure?
Assemble a focus group of people who resemble your target audience. Provide
them with the name of your construct. Ask them how they would define it. Note
how your definition of the construct differs from theirs.
4 Measurement in Education: A Primer on Designing Assessments