December 2021, ScientificAmerican.com 43Paul Ekman
cultural and individual variations in facial ex-
pressions. Many researchers say algorithms can-
not—yet, anyway—consistently read the subtle-
ties of human expressions in different individu-
als, which may not match up with stereotypical
internal feelings. Ekman himself, who worked to
develop early forms of emotion-recognition tech-
nology, now argues it poses a serious threat to
privacy and should be heavily regulated.
Emotion AI is not intrinsically bad. If ma-
chines can be trained to reliably interpret emotions and behavior,
the potential for robotics, health care, automobiles, and other fields
is enormous, experts say. But right now the field is practically a
free-for-all, and a largely unproven technology could become ubiq-
uitous before societies have time to consider the potential costs.in 2018 mark gray, then vice president for people and business op-
erations at Airtame, which makes a device for screen-sharing pre-
sentations and displays, was looking for ways to improve the com-
pany’s hiring process. Efficiency was part of it. Airtame is small,
with about 100 employees spread among offices in Copenhagen,
New York City, Los Angeles and Budapest, but the company can re-
ceive hundreds of applications for its jobs in marketing or design.
Another factor was the capricious nature of hiring decisions. “A lot
of times I feel it’s coming from a fake voice in the back of some-
one’s head that ‘oh, I like this person personally,’ not ‘this person
would be more competent,’ ” says Gray, who is now at Proper, a Dan-
ish property management tech company. “In the world of recruit-
ment and HR, which is filled with the intangible, I kind of wanted
to figure out how can I add a tangible aspect to hiring.”
Airtame contracted with Retorio, a Munich-based company that
uses AI in video interviews. The process is quick: job candidates
record 60-second answers to just two or three questions. An algo-
rithm then analyzes the facial expressions and voice of the inter-
viewees and the text of their responses. It then generates a profilebased on five basic personality traits, a common
model in psychology shorthanded as OCEAN:
openness to experience, conscientiousness, extra-
version, agreeableness and neuroticism. Recruit-
ers receive a ranked list of candidates based on
how well each profile fits the job.
Such software is starting to change how busi-
ness decisions are made and how organizations
interact with people. It has reshaped the hiring
process at Airtame, instantly elevating some can-
didates over others. Gray says that is because the profiling works.
He shared a chart showing that the job performance of several re-
cent hires in sales tracked their personality scores, with employ-
ees who had scored higher in conscientiousness, agreeableness
and openness doing the best.
Machines that can understand emotions have long been the
subject of science fiction. But in computer science and engineer-
ing, human affect remained an alien concept for a long time. As re-
cently as the 1990s, “it was a taboo topic, something undesirable,”
says Rosalind Picard of the Massachusetts Institute of Technology,
who coined the term “affective computing” in a 1995 technical re-
port. “People thought I was crazy, nuts, stupid, embarrassing. One
respected signal- and speech-processing person came up to me,
looked at my feet the whole time, and said, ‘You’re wasting your
time—emotion is just noise.’ ”
Picard and other researchers began developing tools that could
automatically read and respond to biometric information, from fa-
cial expressions to blood flow, that indicated emotional states. But
the current proliferation of applications dates to the widening de-
ployment starting in the early 2010s of deep learning, a powerful
form of machine learning that employs neural networks, which
are roughly modeled on biological brains. Deep learning improved
the power and accuracy of AI algorithms to automate a few tasks
that previously only people could do reliably: driving, facial recog-
nition, and analyzing certain medical scans.INSIDE OUT: Some emo-
tion-AI systems rely on
work by psychologist Paul
Ekman. He argues universal
facial expressions reveal
feelings that include ( from
left ) sadness, happiness,
anger, fear and surprise.