16 January 2021 | New Scientist | 19AN ARTY artificial intelligence
gets inspiration from text
captions to create outlandish
images – such as “armchairs
in the shape of avocados” or
“a baby daikon radish in a tutu
walking a dog”. This hints that it
has some grasp of how language
shapes visual culture.
OpenAI, a company that
has partnered with Microsoft,
developed the system, called
DALL-E. It is a neural network – a
form of AI modelled on a brain –
and is based on the company’s
GPT-3 language model that can
create expansive written works
based on short text prompts,
but DALL-E produces images
instead of words.
“The world isn’t just text,”
says Ilya Sutskever, co-founder
of OpenAI. “Humans don’t
just talk: we also see. A lot of
important context comes
from looking.”
DALL-E is trained using a set
of images already associated
with text prompts, and then
uses what it learns to try to
build an appropriate image
when given a new text prompt.
It does this by trying to
understand the meaning of
the “natural language” text
it is given, then producing an
appropriate image. It builds
the image element by element
based on what has been
understood from the words. If it
has been presented with parts of
a pre-existing image alongside
the text, it also considers the
visual elements in that image.
For instance, if given an
image of the head of a T. rex, and
the text prompt “a T. rex wearing
a tuxedo”, DALL-E can draw the
T. rex body under the head and
add appropriate clothing to it.
The neural network, which
OpenAI described last week on
the company’s website, can tripup on poorly worded text
prompts and struggles to
position objects relative to
each other – or to count.
“The more concepts that
a system is able to sensibly
blend together, the more likely
the AI system both understands
the semantics of the request
and can demonstrate that
understanding creatively,” says
Mark Riedl at Georgia Institute
of Technology in Atlanta.
“I’m not really sure how to
define what creativity is,” says
Aditya Ramesh at OpenAI, who
admits that he was impressed
with the range of images
DALL-E produced.
The model creates 512 images
for each prompt, which arethen filtered using a separate
computer model, also
developed by OpenAI, called
CLIP. This system picks what
it believes are the 32 “best”
works produced by DALL-E.
CLIP is trained on 400 million
images with text found online.
It looks at nouns, verbs and
adjectives, and performs as well
as many image classification
systems when confronted with
images it hasn’t seen before.“We find image-text pairs
across the internet and train a
system to predict which pieces
of text will be paired with which
images,” says Alec Radford at
OpenAI, who developed CLIP.
“This is really impressive
work,” says Serge Belongie at
Cornell University, New York. He
says more research is needed to
look at the ethical implications
of such image creation systems,
like the risk of creating totally
faked pictures. For example,
what if you could fool people
into thinking scenes are real,
not computer-generated.
Effie Le Moignan at Newcastle
University, UK, also calls the
work impressive. “But the
thing with natural language
is, although it’s clever, it’s
very cultural and context-
appropriate,” she says.
For instance, Le Moignan
wonders whether DALL-E, if
confronted with a request to
produce an image of Admiral
Nelson wearing gold lamé
pants, would put the military
hero in leggings or underpants –
potential evidence of the
gap between British and
American English. ❚A FOSSILISED oviraptor found on
top of a clutch of eggs confirms that
some dinosaurs sat on or near their
eggs to keep them warm, like birds.
There is already strong evidence
that some dinosaurs brooded eggs.
Several fossils of adult oviraptors –
bird-like dinosaurs around 2 metres
long – have been found on or
near clutches of eggs. But many
researchers think even small
dinosaurs may have been too
heavy to sit atop their eggs without
damaging them. Some argue that
the oviraptors found on nests died
while laying eggs rather than during
brooding or while guarding the eggs.
Now a team led by Xing Xu at
the Chinese Academy of Sciences in
Beijing has described and analysed
a fossil found near Ganzhou in
China. It consists of the partial
remains of an adult oviraptor on top
of at least 24 eggs, many of which
have embryos inside. Eggs with
embryos have been found before,
but not in association with an adult,
says team member Michael Pittman
at the University of Hong Kong.
The fact that the embryos are at
a late stage shows the adult wasn’t
laying eggs when it died. The team
also looked at the amounts of
different forms of oxygen in the
carbonates in the eggshells and
embryo bones. The ratios of
oxygen-18 to oxygen-16 isotopes
reflect the temperature at the time
the carbonates formed. For two
eggs, this analysis suggests that
the embryos developed at around
body temperature – between
36°C and 38°C (Science Bulletin,
doi.org/fp59). This shows the
eggs were being brooded, not
just guarded, says Pittman.
For the third egg analysed,
which was further from the body
of the adult, the developmental
temperature would have been
between 30°C and 32°C. Some
modern bird eggs develop at a lower
temperature too, says Pittman. ❚Images of a chair and
a radish, with a twist,
made by an AIPalaeontology ComputingMichael Le Page Chris Stokel-Walker400m
images help train OpenAI’s
software to assess its outputAI illustrator draws surreal
images from text prompts
Some dinosaurs
sat on their eggs
like birds do
OPENAI