46 | New Scientist | 9 April 2022
I
HAVE a confession to make: some of
the articles that have appeared in New
Scientist, including ones I have written,
are wrong. Not because we deliberately misled
you. No, our reports were based on research
by respected scientists at top universities,
published in peer-reviewed journals. Yet,
despite meeting all the normal standards of
credibility, some findings turned out to be false.
Science is in the throes of what is sometimes
called the replication crisis, so named because
a big hint that a scientific study is wrong is
when other teams try to repeat it and get a
different result. While some fields, such as
psychology, initially seemed more liable than
others to generate such “fake news”, almost
every area of science has since come under
suspicion. An entire field of genetics has even
turned out to be nothing but a mirage. Of
course, we should expect testing to overturn
some findings. The replication crisis, though,
stems from wholesale flaws baked into the
systems and institutions that support
scientific research, which not only permit bad
scientific practices, but actually encourage
them. And, if anything, things have been
getting worse over the past few decades.
Yet as awareness of the problem has grown,
so have efforts to tackle it. So, how are these
opposing forces faring? Will the efforts to
combat fake science succeed? And how can you
know if the research you read about in New
Scientist and elsewhere will ever make it out
of the lab and start working in the real world?
It is hard to pinpoint when the replication
crisis began, but many people got their first
inkling of it in 2011. That year, three things
happened to ring alarm bells. First, a study was
published that claimed to demonstrate psychic
abilities. Some people apparently experienced
improvements to their memory of word lists if
they were given reminders after being tested.
In other words, they were seemingly predicting
the future. Later that year, a paper showed how
easy it is to get invalid positive results. The key
is “cherry-picking”, producing lots of data and
only using the figures that confirm your
hypothesis, a practice also sometimes called
p-hacking, after one of the terms in a commonly
used statistical technique. A third blow was
the brewing scandal over Dutch psychologist
Diederik Stapel, who was caught simply making
up results. This showed that the much-vaunted
journal peer-review system – where journalsask experts in a field to decide whether a paper
should be accepted – is no guarantee that only
good science gets published.
“That was the year that psychology had this
wake-up call,” says psychologist Amy Orben
at the University of Cambridge. What’s more,
there were growing rumblings that some
seminal psychology results were unreliable. In
2012, researchers failed to replicate the finding
that people exposed to stereotypes of ageing
will walk more slowly when leaving the lab.
The original experiment was a key result in
the field of^ “priming”, the idea that people’s
behaviour is affected by unconscious cues.
Failure to replicate was also the undoing of
another textbook finding, that we can “use up”
our willpower, so if we are trying to resist
tempting food, we find it harder to persevere
with a mental task. Both these once-influential
ideas have now fallen into disrepute.
For some science journalists like myself,
these U-turns and the ensuing rows were
fascinating, but not that concerning.
Psychology has long been seen as less rigorousthan the “harder” life sciences, like medicine or
cell biology, mainly because human behaviour
is so complex, making it difficult to measure
and theorise about. But I was being too
complacent. In the past decade, unreplicable
scientific claims have emerged elsewhere –
even in the physical sciences. It pains me to say
it, but my own beat of medicine and the other
life sciences seem among the most affected.
One of the worst hit areas is early stage
biomedical research: laboratory studies of
potential new drugs tested on animals or in
cells grown in a dish. Such research is regularly
taken up by pharmaceutical firms, who have
the resources to turn it into new medicines.
But it often turns out to be unsound. In 2011,
an internal survey at drugs firm Bayer found
that two-thirds of the leads from university
research were failing to stand up when tested by
company scientists. The next year, biotech firm
Amgen investigated 53 landmark papers and
found that only 1 in 10 could be replicated.
This, for me, was a real shocker. Many people
understand that promising results in mice“ I hate to think how many stories I have
written that gave people false hope”
MONTY^ RAK
USEN
/GETTY^ IMAGES