Evil Empire 111for algorithmic accountability may miss the forest for the trees.
The commensuration at the heart of the SCS is an irremediable
f law. In Spheres of Justice (1983), Michael Walzer defined tyranny
as the domination of one mode of distribution in realms where it
does not belong. It is easiest to see this domination in the realm of
money and illicit commodification. If a billionaire wants to buy a
beautiful handcrafted couch, few mind. However, we rightly object
if the same billionaire wants to buy an election, or be first in line
to receive a donated kidney, or purchase a license to travel at twice
the speed limit. All those rights and privileges are supposed to be
governed by some other standards: democracy, or medical necessity,
or strict equality. Money becomes tyrannical when it overturns the
standards of independent fields.
Algorithmic governance such as the Chinese SCS allows the
government (and its private partners) to consolidate and thus dominate
spheres of reputation that should be decentralized and private. It is
prone to a tyrannical granularity, a spectrum of control—following
Gilles Deleuze’s critique of “control societies”—that is more exacting
and intrusive than older forms of social order. The Chinese SCS both
uses and encourages data feeds, encouraging a spiral of surveillance.
One school aspires to so minutely judge citizens’ lives that not even
evanescent emotions are safe from scrutiny. In an experimental “intel-
ligent classroom behavior management system” described by Business
Insider, cameras scan classrooms every thirty seconds and record “stu-
dents’ facial expressions, categorizing them into happy, angry, fearful,
confused, or upset... [as well as recording] student actions such as
writing, reading, raising a hand, and sleeping at a desk.” There are no
limits to the regimentation a scoring system might produce. Marketed
as “affective computing,” these methods could easily compute optimal
affect, prescribing expressions and thoughts to match them.