Serverless, ReactPHP, and Expanding Frontiers, May 2019

22 \ May 2019 \ http://www.phparch.com

Department of Breaking Changes: Launching PHP 7 in a Highly Available Web World

Due to these disruptions, plus starting
later than expected, the full rollout
for PHP 7.2.x didn’t complete during
regular work hours. Vick decided to act
heroic and push through until about
ten at night to finish up all servers. This
resulted in a late-hours emergency
issue where the web servers reserved
for handling ImageMagick traffic were
under-provisioned with memory and
froze up almost immediately. After
backing out Proxmox and Webzero
changes to those machines, the solution
didn’t get solved until midday the next
d ay.

The smoke cleared, our stomachs
settled, and everyone arrived at work
the next day hopeful. All in all, it was
controlled chaos which resulted in
positive outcomes. There were still
tickets to address, logs to look at, and
issues to discuss. Still, every web
server had successfully become PHP
7.2.x compliant, running the website
for NPR.org, the site’s version of the
Listening Service API, and much of
the backbone for the Carbon API all
successfully migrated over. Months of
planning, discussing back and forth,
disagreement, and meticulous testing
paid off. True, we had some hiccups
during launch but surprisingly none
of the serious ones were PHP related.
Even after it was all done, the team
started thinking about how they all got
there, and how we were going to move
forward. Whatever feedback or insight
our team received, one thing was clear:
there’s no going backward. We made
improvements across the board to our
systems that support NPR.org and
learned much along the way.

Conclusion

Even with the negative impacts of
the project, we came out learning more
from our development team than how to
run projects like this one better. The key
piece of knowledge we took away was
learning there are two competing inter-
ests to overcome when you have a large
codebase with multiple teams making
changes to it, and a team wishing to
alter the infrastructure significantly.
Looking back at many our upgrades,

we realized the work leading up to production rollout had been mostly trial-and-error. We had some idea about the impact on, say, Apache going from 2.2 to 2.4 would be, but had to test it all out in the wild to be one hundred percent sure. While this was going on, the developers were disrupted by the changes in infrastructure like the loca- tion of logs, and that the deployment method was different. There was also the question amongst QA members of whether a test regression is due to code, or due to a regression in the infrastructure. Changing Apache, or even just moving logs around tended to create more questions than answers. With all that we planned to do in addition to upgrading PHP, it was sometimes hard to explain that what developers saw as a flood of code and system changes was actually a logical path to a safer upgrade process. Being as transparent as possible while breaking things tends to go a long way in removing stress from those outside DevOps. Our usage of blog posts, Q&A sessions, department-wide (and not just team-based) retrospectives, and documentation were practices to carry forward with us. Unfortunately, the upgrade to PHP 7.2.x didn’t show any tangible performance results from our New Relic dashboards. iT was not entirely disappointing, however. As with any project with code as old as our website,

improvements are never-ending. The more places we found that needed to be updated or refactored, the more we found that could change for the better. Some of the code that had to be updated for standards also looked to Jason and me like features yearning to be rewritten or deprecated. Cookbook changes led to long discussions of why we host our website code the way we do, conversations that have now led to productive discussions among the wider team. Improving our website code deployment, for example, did cause disruptions. However, those disruptions led to a safer, more stan- dard way of releasing code. The same goes for our changes to Apache and MySQL. The lesson here is change is ultimately disruptive but in our case produced results beyond the upgrade alone. This is why, after it was all over, and although we didn’t see quite as high of a performance impact from the upgrade as we wanted from PHP, we still felt satisfied. It just meant we could look elsewhere for improving our systems: Apache optimization settings for PHP, using improved PHP practices—all made possible by our changes. The question of “would we do it again” is a resounding “Yes.” “Would we do it better?”—with the input we received from our team, absolutely. “Will we ever be done-done with all changes, ever?” Of course not.

Grant is a web developer specializing in DevOps, Jenkins, and anything build related. Hailing from the Washington D.C. area, he works for the Digital Media department at National Public Radio. Previous projects have included working on the NPR One Listening Service, the Reading Service API for NPR, and most recently, refactoring and upgrading infrastructure and code to run NPR dot org on PHP 7+. @jgrantd

Serverless, ReactPHP, and Expanding Frontiers, May 2019

Get our desktop app

Company

Features

Documentation

Resources