42
Question 4: How Does Your Target Audience Think About What
You’re Trying to Measure?
Designing an assessment tool is a social exercise. Question 1 asked, “Who is your
audience?” Your answer to this question is now the key.
At this stage, you’ve defined your construct and its component parts. You’ve
reviewed the literature to see how other researchers have tried to measure that con-
struct. You’ve likely come to the following place: you found some helpful tools that
you think you can adapt and improve upon, for your purposes. Or perhaps you
found nothing. In either case, it’s going to be necessary to write new items for your
assessment. But before you do that, it is beneficial to gather the thoughts of a group
of people who resemble your target audience.
This is not a step taken by all survey researchers. It is inspired directly by the
work of Hunter Gehlbach, Anthony Artino, and colleagues [ 1 , 9 ]. They recommend
forming a focus group: “researchers need to hear how participants talk about the
construct in their own words, with little or no prompting from the researcher.” It is
important to give members of the focus group the chance to speak. In essence, mem-
bers of the focus group should be allowed to form their own definition of your
construct. What does it mean to them? What do they see as its component parts?
Face-to-face interactions are extremely valuable. Technology offers other
options. If done seriously, interviews with a focus group will provide nuanced
feedback.
If you’ve developed careful answers to Questions 1, 2, and 3, your construct
should resonate at least partly with your target audience. Their definition of the
construct should overlap with yours. But in reality you’ll never know what a focus
group is going to say, until you form one. It’s possible that your audience thinks
about your construct completely differently than you do.
If you follow the remaining steps, you’re going to build and refine an assessment.
One way or another, your audience’s understanding of your construct is going to
make its way into your assessment – either through specific feedback in the early
stages or through confused and unclear answers on the assessment itself. A consis-
tent theme throughout the assessment design process is this: it’s better to catch
problems early than to collect bad data later. A focus group can help you prevent
later missteps.
A literature review can help you refine the definition of your construct to fit with
previous research. A focus group can help you refine the definition of your construct
to fit with the ideas of your audience.
Question 5: It’s Time to Start Writing Questions for Your
Assessment, But Do Your Questions Ask What You Mean for Them
to Ask?
You’ve developed an initial definition of your construct. You’ve conducted a litera-
ture review to see how others have defined and measured that construct, and you’ve
C. Hitt