6 March 2021 | New Scientist | 17AN ARTIFICIAL intelligence
that can remember its previous
successes and use them to create
new strategies has achieved
record high scores for some
of the hardest video games
on classic Atari consoles.
Many AI systems use
reinforcement learning, in which
an algorithm is given positive or
negative feedback on its progress
towards a particular goal after
each step it takes, encouraging it
towards a particular solution. This
was used by AI firm DeepMind to
train AlphaGo, which beat a world
champion Go player in 2016.
Adrien Ecoffet at Uber AI Labs
and OpenAI in California and his
colleagues hypothesised that such
algorithms often stumble upon
encouraging avenues but then
jump to another area in the hunt
for something more promising,
overlooking better solutions.
“What do you do when you
don’t know anything about
your task?” says Ecoffet. “If you
just wave your arms around, it’s
unlikely that you’re ever going
to make a coffee.”
To solve this problem, the
team created an algorithm
that remembers all the different
approaches it has tried and keeps
returning to moments in which it
had a high score as a starting point
from which to explore further.
The software stores screen
grabs from a game as it plays to
remember what it has tried,
grouping together similar-lookingimages to identify points in the
game it should return to as a
jumping-off point. The algorithm’s
aim is to maximise its score, and
when it reaches a new high score,
it updates its record of the
particular starting point it used
with a new screen grab from that
part of the game.
Atari games don’t normally
allow players to revisit any point
in time, but the researchers used an
emulator – software that mimicked
the Atari system – with the added
ability to save game states andreload them at any time. This
meant the algorithm could begin
from any point without having to
play the game from the start.
The team set the algorithm
to playing a collection of 55 Atari
games that has become a standard
benchmark for reinforcement
learning algorithms. It beat state-
of-the-art algorithms in those
games 85.5 per cent of the time.
In one particularly complex
game, Montezuma’s Revenge,
the algorithm scored higher
than the previous record for
reinforcement-learning software
and also beat the human world
record (Nature, doi.org/gh5jbc).
Once the algorithm reached
a sufficiently high score, the
researchers used its solution to
train a neural network to play the
game the same way, doing away
with the need for reloading save
states with an emulator.
Peter Bentley at University
College London says the
team’s approach of combining
reinforcement learning with an
archive of memories could be used
for more complex problems. ❚Matthew Sparkes
ATARIAGEBiodiversity
Shadow snake
rediscovered in
Ecuador’s rainforest
AFTER a long hiatus, a mysterious
reptile has slithered back into
the light. Researchers working
in Ecuador’s tropical rainforests
report that a Fugler’s shadow
snake (Emmochliophis fugleri) has
been found alive after 54 years of
absence from the scientific record.
In 2019, conservation biologists
Ross Maynard and Scott Trageser
at The Biodiversity Group in Arizona
were conducting amphibian and
reptile surveys in the Río Manduriacu
reserve in Ecuador’s Andes
mountains. They were on a night
hike when Trageser came across
a small, dark snake wriggling next
to some mossy boulders.
After catching the snake and
getting a closer look, Maynard
noted that despite his normally
robust field identification skills,
he was having trouble categorising
the find. “This I definitely did not
have on my radar,” he says.
The snake was examined
and photographed, then Maynard,
Trageser and their colleagues later
classified it as a Fugler’s shadow
snake – only the second individual
known (Check List, doi.org/fxbd).E. fugleri and its presumed closest
relatives are called shadow snakes
for their dark colouration, nocturnal
habits and secretive nature. They
may have gone undetected for so
long because they are exceptionally
rare or shy, says Maynard.
“I am totally surprised and
excited about the discovery,” saysThaís Guedes at the State University
of Maranhão in Brazil, who wasn’t
involved with the study. She says
assessments of snake biodiversity
have been neglected until recently.
“Snakes are some of the most
secretive and hard to observe
vertebrates out there,” says
Sara Ruane at Rutgers University
in New Jersey, who also wasn’t
involved with the research.
“No doubt there are other species
where rediscovery awaits.” ❚
Jake BuehlerArtificial intelligence
AI smashes video game high score
Learning from previous successes can help an artifical intelligence perform better
An AI beat a human world
record for the Atari game
Montezuma’s RevengeSC
OT
T^ J.^ TR
AG
ES
ERA Fugler’s shadow snake has
been found for the first time
in more than five decades