38 | New Scientist | 22 August 2020
and apply for grants, but not an incentive
to discover the truth. We focus far too much
on rewarding people who have brought in
big grants or published papers in prestigious
journals, which isn’t necessarily getting
us what we want.But there are checks and balances, like peer
review, where independent experts vet papers
before they are published...
It works in some cases, but it’s nowhere near
the filter it needs to be. Some of the worst
papers ever went through the peer-review
system of the world’s best journals.
For instance, the system isn’t set up
in a way that peer reviewers can easily get
raw data. It’s absurd. The people who are
supposed to be checking whether the
analysis is correct rarely see the data that
the claims are based on.
Reviewers are also often rushed. They
themselves are busy scientists. Journals
want there to be only a short time between
articles being submitting and accepted,
so peer reviewers are pushed towards
accepting things really quickly. And journals
only want to accept the flashiest and most
exciting claims.
That’s before you get into any personal
vendettas that some peer reviewers have
for one another. Even if the paper is
anonymous, you can usually tell who
wrote it. If it’s your rival lab, maybe you’ll
give it a very harsh review.Another check in the system is replication.
But that seems to have gone wrong.
I remember being told when I was an
undergrad that a paper should be written in
such a way that anyone could replicate the
experiment. But when people try and do that,
often they can’t even set the experiment up
because the paper doesn’t contain enough
information. When they can replicate the
experiment, they often find results that are
different or not statistically significant.
Scientists are not being fully open
or transparent. They can often be quite
reluctant to give more details when someone
comes along and says, “I want to replicateyour work”, because there’s this culture of,
“Oh, I know best how to do this experiment.”How much published science isn’t replicable?
In my own subject, psychology, replication
was recognised as a major problem from
about 2011. About 50 per cent of studies in
the literature don’t seem to be replicable.
That replication crisis spread into pretty
much every subject that has been looked at.
In most subjects, if you ask scientists, “Do
you think the research is completely robust?”,
a huge proportion say no. But it’s hard to put
a number on it because almost nobody else
has made a systematic attempt to replicate
results. My guess would be that the picture is
less bad in other subjects, but we don’t know.
I think that is itself a demonstration of the
problem. We are not clear what proportion
of papers from biomedicine or chemistry
or physics and so on would be replicated
because replication is not incentivised.
Going back and doing someone’s experiment
again, is, in most cases, not what the journal
system and the university system want.Is the replication crisis getting worse?
I think there’s evidence that things are worse
now than they were 30 years ago in terms of
this obsession with publication. People who
finish their PhDs now are expected to have
some astonishingly high number of peer-
reviewed publications, something like 19.
A few years ago, you’d be expected to have
five or six. The quality of the work inevitably
has to suffer.This a long-standing problem.
Why write a book about it in 2020?
We’ve got to the point where we have
a handle on the issues. We’ve also got a
great deal of so-called meta science, the
science of science. People have spent their
entire careers running large replication
attempts. So I thought it was a good time
to summarise where we are.
A lot of it is still a bit ivory tower,
discussed only within science. When we’re
communicating to the public, we just go
back to “here’s a new exciting paper that tells“ Scientific papers
are seen as this
almost sacred
thing. But a
lot of people
don’t know how
the sausage
is made”