Pasqualeand must ensure that the lucky are not given credit merely for
circumstance. A hospital in a war zone is going to have a higher
mortality rate than one in Beverly Hills. So a risk adjuster tries
to imagine what the average hospital copes with, and then adjusts
any given hospital’s score up or down based on whether it faced
unusual adversity or ease.
Imagine trying to find the “average” for filial piety, one of the
Chinese scoring system’s cornerstones. At what point does a nagging
father deserve a shortened visit from an exasperated son? Should one
of three children have less obligation to spend time with her parents
than a singleton? How much more does a rich person owe to his parents
compared to a poor person? Each question—and countless more—will
need to be answered in a quantifiable and standard way.
Without having even reached this dilemma, however, there is
early anecdotal evidence that the SCS may be failing on its own terms.
For example, a bank may submit false information to blackball its best
customer, in order to keep that customer from seeking better terms at
competing banks. To the extent that the system is a black box, there is
no way for the victim to find out about the defamation.
This basic concern about data quality and integrity undermines
arguments, such as the one posed by the Financial Times, that “Chinese
AI companies, almost wholly unfettered by privacy concerns, will have
a raw competitive edge when it comes to exploiting data.” If guarantees
of due process are limited or nonexistent, how strong can promises of
data quality and integrity be? For true believers in big data, the signal
of ever larger data sources will eventually drown out the noise of gamed
and manipulated feeds; critics caution there is little basis to believe so.
Legal scholars in Hong Kong are already raising concerns
about the SCS’s accuracy and fairness. It is crucial to note, how-
ever, that if reformers focus only on legalistic concerns, their push