239
Learning to Think Like a Psychologist
Best and Worst Revisited: Part II
At the end of the course, students revisit their initial responses to the Best/Worst assign-
ment and reread their explanation for each of their four selections. Then, with the research
methods course in mind, they consider (a) whether they would make any additions or
subtractions in any of the categories, (b) whether their criteria for evaluating problems and
methods have changed, and (c) which specific elements of the course most influenced
their thinking about what makes research valuable. Students’ responses clearly indicate
that they noticed the themes and vocabulary emphasized by the instructor and highlighted
in the textbook, but they also reflect a growing awareness of the specificity of their
knowledge about disciplinary-relevant dimensions and standards for evaluating empirical
research.
Previously chosen “best methods” sometimes turned out to be “packed full of internal
validity problems.” One student reflected on her greater understanding of the value and
necessity of nonexperimental methods: “Now when I look at the method I chose for ‘worst
method,’ it doesn’t seem terrible at all ... I understand now that there is no other way to
investigate this relationship, since running an experiment is unethical.” The majority of
students noted that they would not change the exemplars they had initially selected, but
many recognized that their choices for “best” method contained flaws or were limited in
ways that they had not noticed before.
In place of subjective criteria, such as whether the topic was of personal interest,
they tended to re-evaluate their initial choices by “taking a more critical look at their
methodology and the validity of their experiments.” Some students wrote that the
course “made me emphasize the importance of having a good procedure and opera-
tional definition,” while others made observations about external validity, such as: “I
never really considered it before this class. I never thought about how far the findings
could be generalized and using that as a criteria for a good study.” Another common
outcome was the realization that inconclusive results are not necessarily a sign of a bad
study: “Whether or not a study finds data that is significant or not, if a researcher per-
forms a study correctly, then many things can be learned.” One student wrote that the
research methods course:
has given me a rubric to evaluate research, rather than my subjective opinion alone. ... What
has changed is my ability to articulate why each article is good or bad ... Reading back on
this now, it is very easy to tell that I am talking about confounding variables ... It is these
kinds of advances in terms and vocabulary that have really helped me in the clarity and direc-
tion of my writing lab reports this semester.
In identifying valuable influences, many students confirmed the value of carrying out their
own studies (working in small groups, they carried out a naturalistic observation, developed
and administered a survey, and wrote a proposal for a “perfect” experiment—all on the
same topic or theme), noting “the aspect that has most influenced my thinking about what
makes research worthwhile was that we actually conducted research.” One student’s post-
course self-evaluation embodied many of the points made by other students: