SURVEY RESEARCH
EXPANSION BOX 16
Methods of Improving Questionnaire
with Pilot Tests
- Think aloud interviews.A respondent explains his
or her thinking out loud during the process of
answering each question.
2.Retrospective interviews and targeted probes.After
completing a questionnaire, the respondent explains
to researchers the process used to select each
response or answer.
- Expert evaluation.An independent panel of expe-
rienced survey researchers reviews and critiques the
questionnaire.
- Behavior coding.Researchers closely monitor inter-
views, often using audio or videotapes, for misstate-
ments, hesitations, missed instructions, nonresponse,
refusals, puzzled looks, answers that do not fit any of
the response categories, and so forth.
5.Field experiments.Researchers administer alterna-
tive forms of the questionnaire items in field settings
and compare results.
- Vignettes and debriefing.Interviewers and respon-
dents are presented with short, invented “lifelike” sit-
uations and asked which questionnaire response
category they would use.
Sources:Dillman and Redine (2004), Fowler (2004), Martin
(2004, Tourangeau (2004a, 2004b), van der Zouwen and Smit
(2004), and Willis (2004).
Cognitive interviewing helps us to identify
problems in questionnaires under development
by asking a small number of pretest participants to
verbally report their thinking while answering the
draft questions. It provides a window into respon-
dents’ thinking and problems they face when
answering survey questions. Cognitive interview-
ers probe for additional information about the
process of answering questions. We use this infor-
mation to refine the questionnaire or interviewing
process (see Expansion Box 16, Methods of
Improving Questionnaire with Pilot Tests).
Another related development draws on ethno-
methodology and conversation analysis to study the
interview process as a special type of social
interaction and speech event. These approaches sup-
port the collaborative encounter model and suggest
treating nonstandardized interview behaviors, such
as respondent queries or minor forms of interviewer
feedback (saying hmmm, laughing, smiling) as
opportunities to learn more about the interview.^75
THE ETHICAL SURVEY
Like all social research, we can conduct surveys in
ethical or unethical ways. A major ethical issue in
survey research is the invasion of privacy.^76 People
have a right to privacy. Respondents have a right to
decide when and to whom to reveal personal infor-
mation. We intrude into a respondent’s privacy by
asking about intimate actions and personal beliefs.
Respondents are likely to provide such information
accurately and honestly when asked for it in a com-
fortable context with mutual respect and trust. They
are most likely to answer when they believe we
want serious answers for legitimate research pur-
poses and when they believe answers will remain
confidential. We need to treat all respondents with
dignity, reduce discomfort, and protect the confi-
dentiality of survey data.
A second issue involves voluntary participation
by respondents. Respondents can agree to answer
questions or refuse to participate at any time. They
give “informed consent” to participate in research.
We depend on respondents’ voluntary cooperation
and need to ask well-developed questions in a sen-
sitive way, treat respondents with respect, and be
very sensitive to confidentiality.
A third ethical issue is the exploitation of sur-
veys and pseudosurveys. Because of its popularity,
some organizations and people have used surveys
to mislead others. A pseudosurveyis a survey for-
mat that is used in an attempt to persuade someone
to do something and has little or no real interest in
learning information from a respondent. Charlatans
Pseudosurvey A false and deceptive survey-
like instrument using the format of a survey interview
but whose true purpose is to persuade a respondent.
