106 8 Outcomes and Results
Jorge’s case was relatively straightforward. He had a lot of data, and the tools to
build a variety of graphs and tables that interpreted the data in useful ways, includ-
ing statistical summaries, trends, and behaviour as different variables, were tuned.
Having spent months investigating the data, drawing preliminary conclusions, and
then seeking confirmation (or confounds), he could then choose typical examples
to illustrate his argument. He first listed the data sets and the analyses applied to
set context for these examples, and in some cases could then just summarize the
outcome of the analysis. By having reasonably objective criteria for choosing what
to present, he was able to build a persuasive qualitative description of the work he
had undertaken, and how it supported his original hypothesis.
Dai’s results chapter was, in the end, similar. He had to be exploratory to find
ways to consistently describe the issues encountered by the different scientists he
studied, including assigning issues to scores in the range − 3 to + 3 to quantify their
severity, and, separately, carefully explaining how he had decided which problems
were more important or less important. Once he had identified common themes, and
thus a categorization of the cases, the presentation was straightforward. That is, he
could consolidate his results into tables, and used them as the basis of a discussion
showing how they confirmed his initial hypothesis.
Jackie’s problem was that, fundamentally, she didn’t trust some of the data, and
needed to build her arguments with unusual care. The approach we took was to
build the presentation from small units, each of which represented a single logical
step and which, we felt, could be defended by a simple, unarguable case. We began,
not by looking at the data, but by going back to basics and setting out criteria that a
trustworthy data set should satisfy. In particular, we identified potential sources of
bias or distortion in the data sets, and also tabulated the kinds of factors that might
lead to the results observed in each case. For example, poor academic results might
be a consequence of poor diet—but it might be that poor results lead to depression,
which then leads to poor diet. Some studies attempted to control for such factors
to try and distinguish between these alternatives; other studies were less well de-
signed. One particularly good study noted which students were siblings, so that, for
example, by working from the assumption that siblings usually have similar diet
then it is possible to explore the degree to which differences in performance are due
to other factors.
This foundational analysis allowed Jackie to organize the studies by their
strengths and defects, and undertake meta-analysis (that is, combined analysis of
a group of studies) on the basis of which criteria each study met. She then worked
through the studies re-analyzing the conclusions that had been reached, and showed
that some of these conclusions could be refuted. With the field of explanations
greatly reduced, she could then examine where in the data her own hypothesis was
and wasn’t supported, and ultimately was able to reach nuanced conclusions. In
contrast to the sometimes dogmatic assertions made in earlier work, her results
were thoughtful and reasoned, and she avoided the mistake of making overly strong
claims.
For Don, the initial hurdle was that he had not appreciated that he was trying to
present what were essentially quantitative results; the approach he had taken earlier