47You should provide reviewers with the definition of your construct and the full
list of items you’ve developed. It is best to provide them with a standard reviewer
form, different versions of which are readily available.
Questions to ask your reviewers fall along the following lines. Is each question
clearly written? Is each question relevant to the construct? Are there other relevant
questions I should be asking?
Question 7: What Does Your Target Audience Think About
the Questions You’ve Written?
In response to Question 4, you assembled a focus group that resembled your target
audience. Do this again, but on a one-on-one basis. Earlier in the process, you gath-
ered thoughts on the construct. Here you want to gather thoughts on the questions
you’ve written. This process has been called “cognitive interviewing” – you want
respondents who resemble your target audience to fill out your questionnaire and
then provide details about what they think the question means. This essentially is an
opportunity for your audience to answer Question 5, in their own words: do your
questions ask what you mean for them to ask?
There are various approaches and guides to conducting cognitive interviews. Here
again the work of Artino and colleagues is the most useful in medical education [ 1 , 14 ].
Question 8: It’s Time to Pilot Your Assessment; Do the Numbers
Come Back as Expected?
The first time you administer your assessment should be considered a pilot run.
You’re assessing real learners in the actual setting you’re interested in.
After we have fielded our assessment and collected pilot data, it is time to analyze
the numbers. This will require some expertise in statistics, as well as special software.
In this chapter, we won’t focus on the formulas for calculating various test statistics. It’s
more important to understand the intuition behind each step in your data analysis.
The key property that we will examine in our data, time and again, is variation.
Is there variation in the data we collect? How does our data vary?
We began this exercise because we were interested in a certain construct – a set
of skills or attitudes that we believe varies from person to person. We believe that
measuring this construct is important because we believe the construct correlates
with other important, measurable factors. What explains the variation in our con-
struct? What does variation in our construct help us to explain? Questions like these
can only be answered if the data we collect meets certain conditions.
Do Answers to Each Question Vary?
Our initial data analysis should examine one question at a time. Each question is
designed to help us tell people apart in terms of a given concept. Therefore we are
interested in the variation in responses that we get to our questions.
4 Measurement in Education: A Primer on Designing Assessments