http://www.phparch.com \ May 2019 \ 21Department of Breaking Changes: Launching PHP 7 in a Highly Available Web Worldof pushing the change out everywhere.
The same smoothness carried on to the
production rollout of HAProxy. With
that last preliminary change in place,
and with PHP 7 changes battle-tested
in our staging environments, we were
ready to move forward.
Launch Day
Tensions were high in the morning
planning out the breaking changes
to take place on launch day for PHP
7.2.x. That nervousness would continue
to fluctuate from medium to high
throughout the day as well. At this point,
DevOps had racked up over 170 tickets
in Jira just to complete the project. For
PHP 7.2.x launch day, all of that was
behind us. Everyone in the Sys Admin
room hunkered down, looking at
Rundeck roll through server by server
on production, fully awake and alert.
A partial amount of the alertness was
due to several hiccups in the expected
flow that morning. Ahsan, the main
systems engineer tied to the project,
had to be absent for most of the day
and was spending his morning training
two other engineers to handle provi-
sioning servers. Two other DevOps
team members, Anthony Ghandhi and
Vick Krishnaswamy, were volunteered
to help but had not yet worked with
the new Rundeck setup nor did they
have much practice swapping servers
in staging environments. This resulted
in extra training for things like which
Proxmox server on which to find our
current virtual machines and proper
Rundeck job procedures. With all of
this commotion, the rollout didn’t get
into full swing until around noon, a few
hours later than planned.
Once underway, the operation settled
to an efficient rhythm. Each server went
through the same checklist as with
rolling out the deployment refactor
and for HAProxy. First, Rundeck
called Proxmox to move a fresh VM
image from one system to another
and bootstrapping Chef—the same
process previously used to rollout the
new deployment process. Then the
new code deployment method would
run through its paces. Only when the
machine passed a health check would
it come back online and be added back
to the F5 service. At that point, we had
one out of many machines now serving
PHP 7.2.x code to the world. From my
vantage point, watching both engi-
neers work through each machine andcommunicating on Slack felt like being
in an assembly line.
The day was not without its quick
jump scares, so it also felt a bit like a
thriller. Log output from Rundeck and
the web server error logs caused worry.
Whenever Rundeck converged the
Chef cookbook, red text (Errors) would
appear in the log output. “Known issues”
Ahsan assured us when we ran through
the process with him earlier. The red
text had to do with several VirtualHost
document roots missing their matching
directories in the system. Such glitches
foreshadowed a lengthy cleanup phase,
starting with addressing those improper
or outdated directives. Regardless, the
other engineers and I found ourselves
having to reassure ourselves and
ignore these minor glitches. All the
while, other developers watched the
PHP error logs for possible issues as
various minor warnings popped up
that weren’t detected during QA. The
next big problem was an oversight in
the Chef Webzero cookbook resulting
in Apache directives being left out of
the converged httpd.conf file. These
resulted in making quick fixes at the last
minute to the Webzero cookbook.