Serverless, ReactPHP, and Expanding Frontiers, May 2019

http://www.phparch.com \ May 2019 \ 19

Department of Breaking Changes: Launching PHP 7 in a Highly Available Web World

Updating the Codebases
To find all of the possible PHP migra-
tion issues in such a huge repository, we
leveraged PHPStorm IDEs’ code anal-
ysis tools for known PHP 7 standard
violations. The next step was to run
through our suite of PHPUnit tests to
catch anything else. Once we got proof
of concepts running in our develop-
ment environments, however, we found
more issues. Early on we decided to use
PHP 7.2.x since at the time it repre-
sented the latest and greatest. However,
it also meant many exception elevations
for syntax issues. Going through every
case where we used deprecated func-
tion calls was easy, but finding cases of
“signature mismatch during inheritance”^3
being thrown and halting execution
ended up being trickier to find without
canarying the code somewhere. Exten-
sive QA and scenario planning had to
be done to mitigate as many “gotchas”
as possible. It also meant we had to
update our Composer packages to be
PHP 7.2.x compliant to not break code
depending on them.
From the beginning, a major piece
we knew to work on was replacing the
MySQL connection library, mysqlnd
ms—an extension for high availability
of MySQL servers which uses a
round-robin algorithm to load balance
primary and replica connections. The
open source library is no longer offi-
cially supported for PHP 7 and up^4 , and
a replacement had to be found. There
was and still is a forked version^5 , but it
is only supported by one lone mysqlnd
ms fan who goes by the username
sergiotabanelli. Having one person be
the support staff for a library we heavily
depend on for a site that gets millions
of unique views a day and is the beacon
flag for many NPR fans, didn’t appeal
to our team. After digging into several
options, HAProxy^6 was chosen as the

3 “signature mismatch during inheritance”:

https://php.net/migration70.incompatible

4 PHP 7 and up:

https://phpa.me/mysqlnd_ms-php7

5 forked version:

https://phpa.me/myslnd-ms-php7-fork

6 HAProxy: http://www.haproxy.org

best production-ready replacement in

terms of how we use MySQL with PHP

via round-robin connections and also

for high availability.

Upgrading Apache

While we were in the game of

upgrading PHP we had to upgrade

Apache—yet another epic to add to

the pile. Ahsan Lake, a Sys Admin on

the DevOps team, argued this meant

that we should also change our Chef

workflow for managing Apache and its

mods. Before the upgrade, we compiled

our source code for mods and pack-

ages like PHP. It helped us make sure

our packages were backward compat-

ible with older mods like mysqlnd_ms

which only worked when compiled

locally. This paradigm produced a side

effect where Chef would not recognize

an upgrade because there would be no

version attached to it—just a filename.

Upgrading a package meant deleting

that file before running a Chef conver-

gence. To fix this and allow us to quickly

upgrade machines meant converting to

using aptitude to perform installs and

updates. This change, in turn, meant

an overhaul of the Chef architecture for

our web machines as well as breaking a

lot of older mods. Tyler groaned when

approached with this. The milestones for

the project were slipping out of simple

and going deeper into choppy waters.

He and others on the team were even-

tually won over once Ahsan explained

that with this work, doing an upgrade

would mean a much more straightfor-

ward and—let’s face it—more standard

process. Upgrades and security patches

could be rolled out faster once all web

servers used aptitude over custom-built

Debian packages.

Deployment Hurdles

Then came the issue of significant

downtime while we did the work to

deploy PHP 7 to all live servers. Since

we were using a new cookbook that

had Apache 2.4 and mods built from

packages, it wasn’t technically feasible

to update the cookbook then used by

the web machines. We needed to boot-

strap a new cookbook, which meant we

had to reprovision each web machine

in place. The DevOps team had done

this before, and the fact that we would

be taking machines down one by one in

production wasn’t necessarily a risk.

What was an issue was when the server

came back up, it would be running PHP

7 on a codebase built for PHP 5.6. In the

Jenkins automated deployment process

at the time, each server had a shared

mount tied to one shared directory that

had the website code installed on it.

That is, all servers could only serve one

version of the codebase because they

were all configured to read the code

from one source. Even if we forked

our repo and had two versions of the

web code—one for PHP 5.6 and one

for PHP 7—we could only serve one

version at a time in any environment.

This roadblock produced a lot of back-

and-forth discussions about scenarios.

Would each server get replaced in

Proxmox, totaling some hours, then

the code gets upgraded? Alternatively,

would the team prefer to upgrade the

code, take their chances, and then go

through the machines one by one?

Downtime of any kind meant an impact

not just to the website and it’s users but

also to the consumers and users of the

API and Seamus. Rather than risking it,

the DevOps team decided it was time to

go to the drawing board and come up

with a better deployment method.

Webzero

Once we had researched all of the

work needed to be completed before

upgrading to PHP 7.2.x, we were

finally off to the races. While it at first

seemed overwhelming how much we

planned to do for just upgrading PHP,

we found each piece built on top of

the other. Changing the workflow for

Chef and Apache, a mini-epic spear-

headed by Ahsan, produced a lot of

“wins” down the road for upgrading

PHP safely. Ahsan used an existing

prototype cookbook for web servers

left long ago by Tyler, codenamed

“NPR Webzero.” Webzero was the logic

change for our Chef web cookbooks

where instead of building custom bina-

ries of MySQL Percona, Redis, and