Chris Tankersley

The False Promise of LTS Releases

Posted on 2019-09-01

On August 30th, 2019, Sara Golemon (@saraMG) tweeted out that developers on PHP 7.2 should start planning on their upgrade path to 7.3 or 7.4 since it was about to go into "security-only" mode, which means only security-related patches would be issued for it. If you were on 7.1, it was about to be End-Of-Life'd, which means 7.1 will receive no further patches.

As a "hot take" to this, Sherri W. (@SyntaxSeed) responded with:

I responded to this with my own thoughts:

From there other people joined into a bit of discourse over whether or not a long or short release cycle helps developers. Developers weighed in on both sides.

The Arguments For an LTS Release Cycle

Clients Won't Pay for Upgrades

From Sherri's perspective as a freelancer with 20-30 clients, it is hard to get a client to pay money just because the underlying language has upgraded. We already have problems trying to justify why we should pay for testing, so coming back to a client a year or two after a project is finished just to pay for a non-functionality-adding upgrading can be a hard sell.

I understand the reasoning. I had two clients that were on PHP 5.2 for a very, very long time. When I say "long time," I mean PHP 5.2 had been released in 2006, and these projects were from still in use well into PHP 5.5's release.

The first was a small Bed and Breakfast site that was written into Wordpress. The reservation system that they used was source encrypted with IonCube, a source encryption extension for PHP. The client refused to pay for an upgrade for this plugin, and since it was wrapped in IonCube we could not manually upgrade it. It refused to work on PHP 5.3 or anything higher. I and the original contractor who worked with her could not get it to work.

The second project was a local government project. They had a loaned server that had been paid with through donations that ran Windows 2000 and was hosted at a local library. Since it had been paid and maintained through donations, it was locked to this hardware. The library would only support the machine if it worked with the AD controller. That left us on Windows 2000.

This was around what would be the end of PHP 5.3's life. When 5.4 was released I contacted them about upgrading, especially because Zend Framework 1 was well outdated as well. There was no money for an upgrade at the time.

In both cases, it was a business decision motivated by money that these pieces of software stay at 5.2. They both stayed at 5.2 for a very, very long time.

From Sherri's tweets, she is in much the same boat - many customers just do not want to pay for arbitrary upgrades for infrastructure. You could try and bundle it with new features, but then they may balk at the cost and still decline the project. If releases were slower, they could be tied to more major upgrades.

Business Can Move Slow

Lars Moelleken (@suckup_de) mentioned that sometimes business processes move slower than release cycles. This means that businesses that need stability look toward an LTS release to provide that stability while the business can still provide value for the life of a project.

I have seen this as well. One project I worked on had many different requirements, which included the list of allowed operating systems and software requirements. We had to work with a series of hardware that only worked with specific Linux kernels, and some distributions shipped with versions of libraries we needed (specifically a version of OpenSSL that had some hardening patches applied).

We also had to be very cognizant of changes to the codebase. We had to be careful not to break anything, as loss of functionality could have some very bad consequences for our uses. Getting patches installed for bugs was done in months, not days.

This meant that many of our software stayed on older versions of languages or libraries. Python and PHP were both well out of EOL when I started at the company. When I left, at least PHP was at 5.6, and Python had started a crawl toward Python 3. The underlying OS had not changed, but because the distro did not support rolling upgrades we could do little for in-place upgrades.

We had planned on upgrading all of this, but much of it was tied to a sales cycle and maintenance timeframes. We would have to maintain two versions of the software which was an additional cost for us. It was decided that we would try and upgrade what we could when we had time, and try to push the customer to a new sales cycle which would allow us to switch them over.

We Just Can't

This argument came up during our weekly group get-togethers where we just have a video call and hang out for an hour. The main focus had been a coworker who used to work for an insurance company that did most of their work in Java 6.

When he started, he wanted to use some newer best practices and libraries that would have made their lives easier. There was a lot of pushback from doing this from various sides.

Many of the arguments revolved around either that there was not enough time, or previous consultants had already decided that "Solution X" was a bad fit for the company. A few developers mentioned that some of the new things would just never work for the current solution due to "technical restraints."

In this case, the development team decided that it would be too much work to push forward with an upgrade. Java 6 still worked, so it was better to just continue to deliver functionality with the current setup. Maybe new projects could allower newer setups.

Why LTS is Bad

In all three of the above excuses, actual reasons had been put forth as to why a slower release cycle would be better.

All of them are completely invalid and just excuses for not doing work. I understand the why of each argument. I just do not accept them because, in the long run, they are just causing more work and more pain in the upgrade process. This makes it even hard to justify upgrades because they will cost more and take more time, and are much more prone to failure.

If There Isn't Money Now, There Won't Be In The Future

In the Bed and Breakfast case, she ended up paying for a server all to herself running PHP 5.2 and an older operating system. This also required her to sign off on a security waiver stating that she understood. I am not 100 percent sure she really did understand, otherwise, she could have paid for the new version of the plugin. As a contractor, I had to protect myself.

By the time I had stopped consulting for her, the plugin was not even maintained anymore - she would have to pay for an entirely revamped reservation system. The cost went from what I think was $75 (at the time) for the plugin upgrade to nearly $2,000 to just replicate what the old plugin did.

The Zend Framework 1 application is still in use. I just checked and it was moved to a host running PHP 7.0. The project was never upgraded. In fact, I know this because there were some workarounds I had to do to get Zend Framework to run under Window 2000's version of IIS. The site is now running under Apache httpd, according to the headers, but still has those workarounds. They just moved it. It was a simple application with no private data so I am not really worried from a security standpoint, but no one has bothered to upgrade it.

They have contacted me on-and-off through the years about doing upgrades, but each time the cost is bundled with an upgrade to something newer, and well outside of the price range they want to pay for changes.

If a framework, OS distribution, or language has an LTS release, this increases the length of time between supported releases. This adds additional complexity to upgrades, which increases costs. The increased cost and time are usually seen as a waste because of no new tangible benefit from the increase. Why pay for something that does not add new features or revenue?

Frameworks like Symfony do a good job of having their final releases in a version be somewhat compatible with the new version, making upgrades easier. Even with 3.4 being LTS, the next LTS is 4.4... which if developers are not upgrading now, that means they are waiting for the next LTS, which is going to take longer and therefore cost more, to implement.

If Your Business Moves at a Glacial Pace, That's Your Fault

Saying that a business moves slowly, and therefore release cycles should move slowly, is a farce. I never accept this is a good answer. In fact, Sara Golemon can back this up:

Much like putting off an upgrade because there is no money in the budget, purposefully putting off upgrades leads to the exact same problem - you push an upgrade off until the point it's painful, and the amount of time and money has now increased. Going for Symfony 3.4 to 5.x will not be straightforward. Moving from Ubuntu 14.04 to 18.04 will cause a lot of things to break.

You now are forced to spend more money and time than if you had just kept up with upgrades and updates. Rewriting software from scratch is more expensive than refactoring.

I fought very hard to move to PHP 7 and Python 3 on the one project, and to upgrade the underlying OS. During my tenure, we went from 5.4 to 5.6 with nothing but package upgrades and got those into production without the clients ever noticing.

We did actually get the PHP 5.6 to 7.2 code migration finished (just not put into production) by the time I left. Since my predecessor and myself took great care to use best practices, the actual number of things that phpstan found were fixed in a few hours. Unit tests were added around them to make sure nothing broke.

The Python code was a mess and primarily 2.6, so it was mostly a lost cause. A rewrite was started that included tests upfront. It was not completed when I left, but it was light-years ahead of where the 2.6 code was. The only problem was it meant pulling our lead Python developer off for a few months to do the work, pushing back a release.

I cannot find the tweet for the life of me, but someone brought up the longevity of developers. Since job movement is fairly frequent in our industry, leaving an upgrade for two to three years can mean the loss of knowledge that is required for these upgrades to go smoothly.

By saying that your business processes move slow, and accepting that, you are only making it harder on yourself, or the people that come after you. You are costing your company more money in the long run.

You Can, You Just Don't Want To

This is usually where most developers end up when it comes to legacy code. The code is in such bad shape that it is hard to fix, so there is an unconscious bias to upgrading it. They are worried about having to go to their boss and explain why they need to upgrade and are afraid of being shot down. It is easier to just stay the course and develop features.

As with most of these situations, you are just delaying the inevitable. You are going to have to upgrade someday, and you do not want that someday to be when a massive CVE comes out of nowhere that you have to handle.

Ben Ramsey (@ramsey) does bring up a good point:

Except that you are at the mercy of a maintainer who then decides if a security feature should be backported. As these are backported, they introduce divergences in the codebases, both from the upstream provider (say PHP Internals) as well as have the possibility of changing behavior. There will be some security fixes that cannot be backported because they deal with newer or changed code, so now the maintainer has to decide on re-implementing the fix or leaving it out.

If you hold off because "someone else provides support," or "we just use what is in the repositories," you are giving up and deciding to stay where you are. It will only end in you still being behind and spending more time and money when you have to make an upgrade.

And if you are holding off because you still use mysql_* functions... Stop. Get off your butt and change it. You've literally had years to fix this.

What Can You Do?

First and foremost, start your planning now. Depending on the quality of your application and the age of your infrastructure, you may have little or a lot of work to do. The sooner you start planning the easier time you will have.

Sell The Upgrade

You can start small and start to make some changes right away while you make the business case. Explain to upper management how not doing these upgrades are going to leave you in a bad spot. Here are a handful of things you can use:

  • Finding developers who want to work on older software is always hard. Finding developers who want to work in old languages is harder.
  • Even with backported security you are at a security disadvantage. It takes time for the backports to happen if they actually happen at all.
  • Libraries and tools move on. You will be left with substandard tooling compared to competitors who stay up-to-date.
  • As libraries update, that leaves you to maintain them (especially manually back-porting security patches). This is more work for your developers, and less time you can put toward new features that matter.
  • To do an upgrade at a later date means spending even more time not working on new features. This can put you behind competitors.
  • If you use AWS/Azure, moving to newer PHP versions can get an automatic optimization, meaning fewer servers, which means less cost.

If you find it impossible to sell doing the upgrades, you have two options - do it anyway, or leave.

If you think you can get away with it, or you have the power to do it, go ahead and just do the upgrades. If you really look into it you might find straight version upgrades are trivial, but worst case you will get an accurate estimate on how long the upgrade will take. Remember, the longer you delay, the longer the upgrade will take.

If a company cannot take the time to understand why being up-to-date is a business advantage, then move on to a company that does understand it.

Run Multiple Versions of PHP

I use both Docker and phpenv to handle multiple versions of PHP on a single machine. I can switch between them with some changes and try out my code as I upgrade.

For Docker, you should just need to change your Dockerfile or switch containers out. It will depend on how your setup is configured. A huge selling point of Docker is the ability to swap out containers, so if you are using Docker (and honestly, if you are using Docker but can't upgrade PHP... WTF?!), this should be fairly easy.

For locally installed PHP, I love phpenv. It allows you to have multiple versions installed at once, and has directions for setting up both PHP-FPM and Apache httpd.

Laravel Homestead is one other option. It is a vagrant-based virtual machine with PHP 5.6 through PHP 7.3. Even if you do not use Laravel, you can throw a normal PHP application in there as well and start switching PHP versions.

PHP tries very hard to keep backward compatibility, so unless you are using a deprecated feature like mysql_* functions your app might just work out of the box.

Figure Out Code Changes

Look at upgrading your PHP version first. If you are on PHP 7.0 or 7.1, great! PHP does an awesome job at adhering to SemVer, so there should be little work you need to do for the minor versions. The PHP manual contains release and migration notes for each version since 5.0. Read the migration notes for each version:

Ignore new features and focus on any changes you need to make.

Tools like phpstan can check your code against PHP 7 and make suggestions on things you will need to change. As I mentioned before, I took our PHP 5.6 codebase and ran it against PHP 7.2 and only had a handful of things to change. You may have more, but it gives you a detailed list of what needs to be fixed.

Actually Upgrade

Most mainline distributions have good maintainers that keep PHP up-to-date. These packages go through the same process for inclusion as any other package, so convince your systems or operations team to update. If they provide pushback, you will need to come up with a good business reason (the packages are official, safe, newer PHP has better security support, is faster, etc). Since they are official repositories, it is not that hard for them to get added into a system. It's not like the operations team needs to compile it themselves.

For Ubuntu/Debian there is the set of packages from Ondřej Surý, available at https://deb.sury.org/. He has worked for years to provide high-quality Debian packages, and all of the PHP packages are either directly from him or based on his packages.

On RHEL/CentOS/Fedora, you have packages from Remi Collet, available at https://rpms.remirepo.net/. He maintains packages for core PHP as well as a bunch of extensions, for various versions of PHP. As with Ondřej, Remi is the package maintainer for Fedora, so these are as official and safe as you are going to find for RPM-based systems.

Don't Delay, Start Now

I hope at this point I have convinced you why something as nice sounding as LTS releases are not as cozy and safe as they make themselves out to be. You are sacrificing time and money later for perceived stability today.

A project that stays up-to-date, and puts into processes that help update in real-time, will be able to stay competitive longer. If security is something that is an ingrained part of software development, why isn't upgrading? Like security, upgrading isn't something you bolt-on, or do later.

Stop making excuses, and start upgrading.

Comments