The Wall Street Journal - 16.03.2020

(Ben Green) #1

THE WALL STREET JOURNAL. Monday, March 16, 2020 |A

Test, Test

And Test Again

Experimentation Works
By Stefan H. Thomke
(Harvard Business Review, 272 pages, $32)

The Power of Experiments
By Michael Luca and Max H. Bazerman
(MIT, 211 pages, $29.95)

BOOKSHELF| By David A. Shaywitz


our centuries ago, Francis Bacon published “Novum
Organum,” a treatise describing the use of
experiments to build knowledge and “put Nature to
the question.” The application of this method to business
has gained new urgency with the explosion of online
commerce and the recognition that small tweaks can
generate enormous profits. As Stefan Thomke notes in
“Experimentation Works,” tests at Microsoft in 2012
revealed that a tiny adjustment in the way its Bing search
engine displayed ad headlines resulted in a 12% increase
inrevenue, translating into an extra $100 million annually
for the company in the
U.S. alone.
The comparison of two
potential options—known as
A/B testing—is now routinely
baked into the development
of customer-facing software,
Mr. Thomke reports.
Microsoft, Amazon, Facebook
and Google “each conduct
more than ten thousand online
experiments annually,” he
writes, adding that even
companies without tech roots
(Nike, State Farm) run trials
like this regularly. The tests might
evaluate, say, the components of a
website—style of font, color of background,
shape of buttons, choice of words—and continuously adjust
them based on user response. Empiricism is critical,
explains a data scientist: Without it, “our
predictions of how customers behave are wrong nine times
out of ten.”
As much as Mr. Thomke, a Harvard Business School
professor, believes that “allbusinesses should be
experimenters,” he wisely observes that “not all innovation
decisions can be tested.” A/B testing may not be the best
way to evaluate a completely new product or a radically
different business model, he concedes, but the approach is
the ideal driver of small changes. Though we celebrate
disruption, Mr. Thomke urges companies to “tap into the
power ofhigh-velocity incrementalism,” explaining that
“most progress is achieved by implementing hundreds or
thousands of minor improvements.” He points to a study
that attributes 77% of economic growth to improvements in
existing products and notes that the structured system of
incremental improvements that Lego implemented following
its near-bankruptcy in 2004 drove 95% of annual sales and
helped restore the company to profitability.
Experimentation, Mr. Thomke observes, is premised on
uncertainty, admits the possibility of imperfection, and (if
done properly) is associated with a high failure rate, since
most experiments yield negative results. This mind-set can
be hard to square with corporate cultures that value the
projection of confidence, the pursuit of consistency and the
elimination of waste. The push for “standardization and
efficiency,” Mr. Thomke says, “can get in the way of
learning.” Yet “outsized discoveries,” argues Amazon’s Jeff
Bezos, “are highly likely to require wandering.”

In “The Power of Experiments,” Michael Luca and Max
Bazerman, also professors at Harvard Business School, note
that the experimental method can be applied to broader
social goals. (One wonders if the near-simultaneous
publication of these books reflects an A/B test of its own.)
The authors are especially intrigued by “nudge units,” teams
tasked with applying the tools of behavioral economics to
populations. In 2011, for instance, a British team showed
experimentally that, by adding a single sentence to a letter
mailed to delinquent taxpayers—“By now, 9 out 10 people
in your town have paid their taxes”—it could recover more
revenue. Similarly, the way decisions are framed makes a
difference. Asking car drivers whether they want to “opt out”
of organ donation—instead of requiring them to “opt in”—
leads to more organ donors.
The authors also illustrate how experiments can reveal
otherwise hidden biases. Inquiries on the home-rental
marketplace Airbnb from mock users with African-
American-sounding names were 16% less likely to get a
response than identical profiles with white-sounding names.
The publicity that these results generated prodded the
company to confront the problem, using experiments to
guide its progress.
Experiments have been applied to political campaigns as
well. Yale researchers, for example, showed that in-person
canvassing was far more effective than phone calls or
postcards in motivating citizens to get to the polls. Todd
Rogers’s pioneering use of experiments to fine-tune voter
outreach was viewed as a critical factor in Barack Obama’s
2008 election, Messrs. Luca and Bazerman remind us, and
inspired Sasha Issenberg’s 2012 book “The Victory Lab:
The Secret Science of Winning Campaigns.” Similarly, the
sophisticated use of Facebook’s platform to optimize
messaging helped drive Donald Trump’s success in 2016.
Like Mr. Thomke, Messrs. Luca and Bazerman balance
their passion for experiments with a recognition of its
limits. Consider a recent, well-designed study—conducted
by University of Pennsylvania professors Katherine Milkman
and Angela Duckworth—with the worthy goal of nudging
participants to take more physical exercise. While the initial
data looked encouraging, according to Messrs. Luca and
Bazerman, “behavioral interventions that led to short-term
gains were less effective when looked at over a multiple-
month span.” As Ms. Duckworth succinctly concludes:
“Behavior changes are really *#$@ing hard.”
This observation applies to organizations too, of course.
Experiments are often unsettling to contemplate and
difficult to execute—and their conclusions hard to accept.
Still they are a powerful tool for curious teams with the
confidence, and humility, to embrace the Baconian

Dr. Shaywitz, a physician-scientist, is the founder of
Astounding HealthTech, a Silicon Valley advisory service,
and an adjunct scholar at the American Enterprise Institute.

The comparison of two options—known as A/B
testing—is baked into product development,
along with other empirical trials and judgments.

It’s Dangerous to Test Only the Sick


resident Trump says 1.
million tests for the
novel coronavirus will
become available this week.
That’s welcome news. But offi-
cials are about to make a mis-
take. The president said testing
will be limited to people who
believe they may be infected.
“We don’t want everybody tak-
ing this test, it’s totally unnec-
essary,” he said.
This would make sense if
there were a cure. Without
one, this strategy won’t curtail
either the epidemic or the anx-
iety associated with it. We will
continue to bleed billions of
dollars in economic costs from
disruption of normal life.
Testing has two purposes
apart from diagnosing individ-
ual cases. The first is to obtain
accurate information on the vi-
rus’s infectivity and mortality
rates. If the true rates for the
coronavirus are similar to
those of the flu, then it isn’t
necessary to shut down the

global economy and lose tril-
lions of dollars. But if they’re
much higher, drastic measures
are imperative.
Testing only sick or symp-
tomatic patients will not get us
to the truth. To see why who
we test matters, consider the
flu. Its mortality rate is around
0.1%—meaning that ofevery-
one infected with the flu,

tested or not, 1 in 1,000 die of
it. If we only tested people
who are hospitalized with flu-
like symptoms, the mortality
rate jumps 75-fold. Similarly
with the coronavirus, testing
only sick and symptomatic
people will result in an overes-
timate of mortality, which
would heighten fear and anxi-

ety and worsen their economic
The way to learn the truth
is to test a random sample of
the population in major cities
with an outbreak. Random
testing would reveal the true
mortality rate and also how
many people have the virus, an
important factor in determin-
ing its infectivity. Authorities
need to start conducting ran-
dom testing now, with statisti-
cians in the coronavirus com-
mand center guiding the
design. If the infectivity and
mortality rates turn out to be
similar to those of the flu, this
approach could avert billions
of dollars in economic loss and
calm public fears.
The second purpose of test-
ing is to avert spread by isolat-
ing those who are infected. In
this regard, it is unclear that
relying exclusively on people
who are volunteering for the
tests makes sense. These are
probably people who are ex-
hibiting symptoms and heed-
ing public-health messages to

isolate themselves at home, as
they would do for seasonal flu.
A study of flu-vaccine strate-
gies (of which I am a co-au-
thor) shows that self-selection
doesn’t get at the high-risk
A good strategy would be to
combine drive-thru tests with
targeted testing of high-risk
populations to try to catch
people who are unwittingly
spreading coronavirus.
The response to the coro-
navirus has already cost bil-
lions of dollars due to shut-
downs and other economic
disruptions. A global reces-
sion may be in the offing. That
is a big price to pay as insur-
ance against a risk that is not
well understood. We don’t
need to accept all the fear and
anxiety as inevitable. Proper
statistical testing can give us
the answers.

Mr. Sood is a professor at
the University of Southern Cali-
fornia’s Schaeffer Center for
Health Policy and Economics.

By Neeraj Sood

Random sampling
is essential to learn
the truth about virus
spread and deadliness.


The junk-
nanced $6.
billion buyout
of United Air-
lines col-
lapsed on Fri-
day, Oct. 13,

  1. The
    stock market
    dropped 191
    points or almost 7%, equiva-
    lent to a 1,500-point drop to-
    day. Scary.
    Rod Berens, head of re-
    search at Morgan Stanley at
    the time, called a department-
    wide meeting after the market
    closed. I dreaded going. I had
    a list of tech stocks I was rec-
    ommending that got killed
    that day. When I showed up, I
    noticed several cases of cham-
    pagne—a weird way to cele-
    brate the loss of wealth. But
    why not? I grabbed a glass; it
    was Friday.
    And then my boss headed
    toward me. Uh oh, I thought,
    he’s going to fire me. Instead,
    he smiled, gave me a high-five
    and said, “Congratulations,
    now it’s your turn,” and
    clinked my glass. Talk about
    confused—one of the worst
    days in my professional life
    and I get congratulated?
    I didn’t see it at the time,
    but he was dead right. Eras
    change, sometimes overnight.
    Over the next decade, the
    stock market shifted from
    junk-bond-fueled buyouts, a
    dominant Japan and the Cold
    War to highly valued emerg-
    ing technology companies,
    ending with the dot-com

Crisis Means a New Business Era

The current market turmoil
tells me a new era is breaking,
so question everything. Will
cable, energy, mobile and so-
cial media ever come back?
And if not, what’s next?
Well, the knee-jerk reac-
tions will come first. Most
think the 2003 SARS epidemic
in China ushered in that coun-
try’s era of e-commerce, but it
was going to happen any-
way—the crisis only acceler-
ated it. So be wary about talk
of robot and drone deliveries.
It may come to that if the vi-
rus spreads, but the econom-
ics still seem far out.
Will energy stay cheap for-
ever after this week’s devasta-
tion? I doubt it, but the econ-
omy can finally benefit from
fracking’s cheap natural gas.
I’d bet so-called clean and re-
newable energy was set back a
decade by having to compete
with lower prices. Cheap fossil
fuels may also push back any
new adoption of carbon-free
nuclear energy.
More interesting is the
emptying of countless college
campuses, sending students
home. Classes will be online-
only until further notice.
Smart. But at some point par-
ents will surely ask, “Why
again are we paying 78 grand
a year?” Is the end of univer-
sities far behind?
Similarly, lots of companies
are telling employees to work
at home. Will an era of tele-
commuting and no rush-hour
traffic finally arrive? Maybe
not. Recall that early in Ma-
rissa Mayer’s tenure as Yahoo
CEO, she banned working
from home because so many

people were, as they say, mail-
ing it in.
The end of China’s domi-
nance is certainly coming. No
one will ever again concen-
trate manufacturing in China
alone. Vietnam and other
countries with low-cost labor
will benefit. Maybe this is Af-
rica’s moment. Related: A
louder globalization backlash
may arise again—but since
consumers like cheap goods, it
will flame out.

Another observation: Inter-
est rates and the Federal Re-
serve may be increasingly ir-
relevant. The Fed’s job seems
to be to ensure the availability
of Treasurys—mostly, as we
wrote last September, as col-
lateral for the repo market,
which finances an increasing
share of global trade. The
$500 billion announced
Thursday is encouraging.
What about mobile and
cloud computing, and even the
stock market and its trillion-
dollar valuations? It’s worth
asking, as venture capital and
private equity using cheap
debt are keeping companies
private longer, or forever. Oth-
ers think it is value stocks’
turn to shine, but that usually
requires a period of inflation
that I certainly don’t see com-
ing. Quite the opposite.

No, growth will still rule,
but with a different set of
leaders. In the bio world, DNA
sequencing and Crispr gene
editing are starting to ramp
up. Health care will be trans-
formed by new ways to detect
and treat cancer and other
ways to cure previously incur-
able diseases like sickle-cell
In the high-tech world, mo-
bile seems tired. Apple’s hot-
test product is AirPods Pro,
those Star Wars-inspired
white sticks hanging out of
everyone’s ears. That’s an ac-
cessory, not an innovation.
Here’s hoping for some knock-
your-socks-off new mobile
products. Note also that we’re
only about a third of the way
into the cloudification of en-
terprises. And we’re only be-
ginning to master machine
learning and artificial intelli-
gence, with their ability to
find patterns that humans
can’t. I think the next tech era
will be driven by implementa-
tion of AI-infused systems
into every business.
Finally, I asked the man
himself, Rod Berens, whom
he’d high-five today. He says
that the past 30 year’s tech
abundance means the develop-
ing world’s billions will finally
see productivity improve-
ments and attract an increas-
ing share of investment. That’s
probably right.
New eras are notoriously
hard to predict. So instead of
focusing on which cities are
quarantined, start thinking
about what’s next. Very few
investors do.
Write to

Cheap energy, remote
China will shape the
long-term economy.

By Andy

If it’s true the
novel corona-
virus doesn’t
like warm
weather, it
could explain
why Latin
America has
yet to report
a large out-
break. Much
of the region
isn’t tropical, but most of the
temperate zones in the North-
ern Hemisphere have mild
winters, and in many places
spring is already arriving.
Too bad the annual thaw
can’t save the Americas from
the economic consequences of
a pandemic, exacerbated by a
decade of easy money. The
meltdown in global stock mar-
kets suggests something
deeper than a corona recession
is at work. And as a version of
the old saw goes, when the U.S.
gets sick, Latin America gets a
lot sicker.
The explanation for the
slowdown in the real economy
is Covid-19, the disease caused
by the novel coronavirus.
Transmission of the illness
isn’t well understood, the se-
verity of individual cases var-
ies greatly, and public-health
authorities are having trouble
getting a grip on how many
people have been infected.
The virus killed a 34-year-
old doctor in Wuhan, China,
who attempted to warn his
country about it. That author-
ities in Beijing initially tried
to silence him raises questions
about China’s claim that it has
quickly contained the dis-
ease’s spread. Transparency
isn’t one of the Communist

Economic Flu Stalks Latin America

Party’s strengths.
Yet business investment in
the U.S. was already trending
down before the virus hit, and
we’re overdue for a recession.
Even the pandemic doesn’t ex-
plain the severity of the stock-
market downdraft last week.
Yes, there is the madness of
crowds, but that too is an un-
satisfying narrative. Some-
thing else is going on.
That something is fear
about U.S. financial-system
stability. The immediate con-
cern for central banks is a li-
quidity squeeze that para-
lyzes lending and brings
down healthy institutions.
Walter Bagehot’s sage advice
in his 1873 book, “Lombard
Street,” was that to avoid
panics, central banks, as lend-
ers of last resort, need to
make liquidity freely available
to solvent institutions, but at
a premium.
On Thursday the Federal
Reserve announced $1.5 tril-
lion in short-term funding to
keep financial markets from
seizing up, though there’s no
penalty rate. A shortage of
dollars in Asia last week hit
Japanese and Korean bonds.
Access to swap lines should
alleviate some of that pres-
sure, and Mexico’s central
bank also has swap privileges
at the Fed. Other central
banks in the region don’t, and
they will feel the pinch.
A greater concern is
whether the U.S. financial
system is facing a solvency
crisis, i.e., too many loans go-
ing bad and pulling one or
more banks under. The Fed
doesn’t reassure with its an-
nouncement Sunday that it

will not only cut its bench-
mark interest rate to near
zero but also buy $700 billion
in Treasury and mortgage-
backed securities.
The purchases are ostensi-
bly to ensure “the flow of
credit.” But this “quantitative
easing” was last seen after the
2008 financial crisis, when
poorly managed financial in-
stitutions were desperate to
avoid insolvency and repair
their balance sheets.

The worry that banks
might be in trouble again is
reflected in markets. Bank
stocks rallied Friday after the
Fed promised to intervene,
but bank equity prices were
diving more than the broad
indexes most of the week. On
Wednesday the Journal re-
ported that “the cost of insur-
ing against default on Citi-
group debt for five years rose
to $115,000 annually from
$40,000 this week, according
to FactSet.” The price re-
mained “well below the pan-
icked pricing seen in 2008,”
but the surge indicated some-
thing amiss.
The benign explanation is
an expectation that bank prof-
its will underperform as inter-
est rates swoon. But the qual-
ity of loan portfolios is also a
consideration. So is the ade-
quacy of the stress testing

banks have faced since the
last crisis.
In an April 2013 speech in
Basel, Switzerland, Thomas
Hoenig, then vice chairman of
the Federal Deposit Insurance
Corp., warned deposit insur-
ers that the practice of risk-
weighting assets was creating
an “illusion that banking or-
ganizations have adequate
capital to absorb unexpected
losses.” If that’s true it would
go a long way in explaining
fear about banks’ exposure to
this downturn. There are also
the uncertainties of the hu-
mongous and nontransparent
interest-rate swap market,
where major banks are coun-
terparties to each other.
If the financial system is
still shaky after a decade of
Fed subsidies, it implies that
absurdly low interest rates and
quantitative easing have been
counterproductive. Grandma
has been eating cat food while
the banks remain fragile. This
is bad for those of us who pre-
fer liberty to socialism.
Former Salvadoran Finance
Minister Manuel Hinds writes
in his forthcoming book, “Our
1776 Moment: A Defense of
Liberal Democracy,” that the
damage from the bailout in
2008 went “much beyond eco-
nomic losses. The crisis and
its solution destroyed the faith
that people had put in capital-
ism and liberal democracy.”
Enter Bernie Sanders.
The U.S. may yet fight off a
lurch to the left. But the rip-
ple effect in Latin America,
where collectivists perpetu-
ally rail against markets, is
Write to O’

The moral hazard of
Fed subsidies leads to
another crisis and
empowers socialists.

By Mary

Free download pdf