38
- Once you start writing questions for your assessment, do your questions ask
what you mean for them to ask?
Choose your words very carefully. Make questions as straightforward as pos-
sible. The quality of your data depends on it. - What do the experts think about the questions you’ve written?
Find experts in the area you’re focused on. Ask them for feedback on your
assessment items. Determine if any additional questions are needed and if all of
the questions you’ve written are relevant. - What does your target audience think about the questions you’ve written?
Again collect the thoughts of people who resemble your audience. Let them
see the items and questions you’ve developed. - It’s time to pilot your assessment: do the numbers come back as expected?
Check the variance, consistency, and validity of your data. This requires some
statistical skills. Just as importantly, it requires intuition regarding the statistical
tests needed. - Is your assessment interesting and useful to other researchers?
Make your assessment available to other educators and researchers. Present
your results at professional and academic conferences. Attempt to publish your
results in an academic journal.
Question 1: Who Do You Want to Assess?
Who are the learners you want to assess? A simple answer might be “residents” or
“surgical teams” or “nurses” or “surgeons.” Provide as much detail as possible. The
more information you have about your learners’ backgrounds, the better.
At this stage in the process, you may not have available all of the background
information you hope to eventually have. Make an educated guess about the charac-
teristics of your audience – this will be important later. Also, take notes of back-
ground information you’d like to have. When you assess your students, you can
include a form that gathers background information.
The level of background information you’ll want to collect is going to be deter-
mined by the skills you’re attempting to assess. There is some information, how-
ever, that you’ll almost certainly want to collect no matter what the context:
educational background, job title, and professional experience.
Question 2: What Are You Trying to Measure?
No matter what we’re trying to assess – skills or knowledge or attitudes – we have
the same problem. We are trying to measure something we can’t directly see, some-
thing that is complex.
In assessment terminology, the thing you are trying to measure is called the “con-
struct.” It’s a bland word but it fits. Consider a simple definition: a construct is an
C. Hitt