Evil Empire 109been blacklisted from travel. The SCS may also eventually rely on peer
scoring, a method patented (though not implemented) by Facebook
in the United States. That is, if an activist criticized the government
or otherwise deviated from prescribed behavior, not only would her
score drop, but her family and friends’ scores would also decline. This
algorithmic contagion bears an uncomfortable resemblance to theories
of collective punishment.
The real governance innovation in the SCS, however, is what might
be called a “spiderweb” or “ripple” effect: a misdeed in one area of life
can have consequences far beyond it. For example, imagine a small firm
pollutes a river, and environmental regulators find out. In old gover-
nance systems, an administrator or court would assess the infraction
and decide whether to impose civil or criminal penalties on the firm’s
owner or managers. While the outcome could be devastating for an
executive, it ended there.
The new algorithmic governance of the SCS would make the pol-
lution a problem throughout the executive’s life. He might be denied
the right to fly or to use the Internet—or he may need to pay more
to do so. His children might be denied a place in preferred schools.
Travel abroad could be rejected outright. Loans might cost more, and
access to social services restricted. The SCS’s stated aim is to enable the
“trustworthy to roam everywhere under heaven while making it hard
for the discredited to take a single step.” Every “misdeed” is another
step toward the internal exile of “discredit.” Every merit expands the
scope of one’s world.
Yet this system also relies on someone defining each and every
misdeed and merit. Such a system, as it expands, will eventually
run into classic dilemmas of metrics. For example, every objective
scoring system must embrace risk adjustment. It must give com-
petitors some compensation for adversity beyond their control