45answer with the whole operating room team in mind or with respect only to other
surgeons? to other surgeons? Perhaps she would be thinking about her fellow
residents or her friends outside of work. The item above gives no guidance at all
on this issue. Unless you know what your reader has in mind with respect to her
peers, you do not know how to interpret her answer.
One solution to this problem is to replace “my peers” with a more specific term,
such as “my co-workers in the operating room.” Our items would then read, “My
co-workers in the operating room are caring,” “My peers in the operating room
are hardworking,” and so on.
Another alternative is to offer a prompt before the series of items that ask about
one’s peers: “Please answer questions about your co-workers in the operating
room. The questions below refer to them as your peers.” A problem with prompts
and vignettes is that readers sometimes don’t read or remember them.
Nevertheless, it’s better to include a prompt at the top of the page than to leave
readers on their own to interpret a vague word like “peers.”
We must also scrutinize syntax. The general rules of writing apply. Survey items
should be simple and direct. As sentences get longer, they get harder to follow.
Shorter isn’t always better, but words should be added only when necessary. Adding
words, like we did above when replacing “my peers” with “my co-workers in the
operating room,” comes at a cost.
Avoid questions that are too wordy for your audience: When building an assessment
for schoolchildren, an unofficial rule is that a question should be readable to
students three–five grade levels below the target audience. There are methods for
calculating readability. Word processing software can provide readability statis-
tics for highlighted text, or formulas for calculating readability can be found
online. The first sentence in this paragraph, for example, is written at a 12th
grade level, according to its score on the Flesch-Kincaid readability test. Knowing
your audience is key when writing items (see Question 1). If writing a survey for
a college-educated audience, a 12th grade reading level is about the right read-
ability target, if possible. The item “My co-workers in the operating room are
hardworking” is written at 9th grade reading level.
So far, we have adjusted almost every word of our survey item. We haven’t yet
discussed how to structure response options.
Avoid answer choices that are too narrow: Our example item offers the response
options “always,” “sometimes,” and “never.” Two of those words are very clear,
and one is very vague. When using Likert-type items, a certain amount of ambi-
guity is unavoidable. The consensus in the literature is that a Likert-type item
should offer at least five response options (e.g. [ 1 , 2 , 8 , 9 ]). For our example item,
a considerable improvement would be “always,” “most of the time,” “some-
times,” “rarely,” and “never.”
4 Measurement in Education: A Primer on Designing Assessments