CentOS 6 - Great, but for how long?

Date July 11, 2011

I think I ruffled some feathers yesterday with a twitter post, where I put up the following message:

I got a lot of replies from people who were understandably curious why I would say that, so I'm writing this blog post as a justification. Hopefully by the end of it, you'll see why I'm concerned.

To begin with, let's make sure we're all on the same page.

In the dark ages...

Linux development began in 1991 by Linus Torvalds as a clone of the Minix kernel. As you may know, and contrary to popular usage, "Linux" refers to the kernel, not to the operating system itself. The kernel acts as an interface between the hardware and the "user land" software, as well as performs memory management, and several other important, but low level functions.

The "operating system" that uses Linux has almost always primarily consisted of core UNIX-like utilities provided by GNU, as well as a widely-differing array of other software, depending on who was deciding what to put in.

Development of the Linux kernel (and in most cases, the other freely available software that came packaged in it) were developed almost entirely by volunteers who's only payment for their time was having more free software available to use.

Patterns Emerge

As time went on, several arrangements of these operating systems became more widely used. These individual arrangements of software were called "Linux distributions", since the underlying kernel was Linux, and it was just the user-land software being distributed was what varied (as well as certain configuration changes, such as where certain files and directories were stored).

The most widely used distributions were Slackware, Debian, and RedHat. You probably recognize these because they're still in active development, albeit with certain modifications in some cases.

As all of the software was Free and Open Source, the distributions themselves didn't cost anything either, although there were businesses which sold Linux distributions on media, since in the mid-1990's home bandwidth was usually limited to 50Kb/s or so, and downloading CD images was time prohibitive.

Beginnings of Support

In August of 1995, Red Hat announced the release of Red Hat Commercial Linux 1.1 which cost $39.95 and included support with installation (although additional support was available, presumably for an additional cost). This was something of a novel idea in the Linux world, and a good number of Open Source advocates were not happy with the idea of spending money for free software.

In any event, this opened the doors for wider use of Linux in corporations which required software to have vendor support. This isn't to say that Linux usage exploded because of this move, however it allowed expansion into a new niche, and served as the thin end of a wedge for Red Hat to charge for Linux distributions in exchange for support.

The Modern World

Flash forward to today, and there are several commercially available Linux distributions, many of which are based on Redhat and Debian. Red Hat has a widely successful Linux release called Red Hat Enterprise Linux (hereafter referred to as RHEL). This is the distribution that I'm primarily concerned with at the moment, because it is essentially the standard that the company I worked for based our infrastructure.

My story

Redhat Enterprise provides a stable environment with corporate support. New releases and software updates are thoroughly tested, and reliability is paramount. It is also not cheap. That being said, the few times I had to call support, they solved my issues quickly and without any data loss.

If you are a corporation and are capable of paying for enterprise support (and you have a business case for it), then you could do worse than buying Red Hat Enterprise Linux. That was the reason that I initially bought several versions for my infrastructure. As it turned out, though, the support costs wore on us over time, and we began to investigate other options.

It's hard to switch out of an existing ecosystem, especially after spending any time in it. You start to create things with the assumption that the environment is a certain way (this was before I learned about the environment abstraction capable with things like puppet and CFengine). I had already made the switch when transitioning between Slackware and RHEL, and I was not eager to do it again.

Imagine my elation when I found out that there was a freely-available clone of RHEL, and that I could essentially swap out one installation for another without changing much (if anything) in my scripts! Yes, of course, I wouldn't have support, but we'd determined that there was a business reason for not paying the rates RH asked, so I wasn't concerned about that (and at this point, I'd been a Linux jockey for about 10 years, so I was fairly confident).

That was how I felt when I found CentOS. It is essentially a binary-compatible release of RHEL. Because the software comprising RHEL is Free / Open Source Software (FOSS) under the GNU license (mostly, I believe), any changes that Red Hat makes must be distributed as well. So the makers of CentOS take the source code, remove any Red Hat logos or other copyrighted material, then repackage it as the Community ENTerprise OS.

The idea is great, and I jumped in fully. When I left my company, almost every server was running CentOS. The installation is rock-solid, and I never had a situation where I even wished I had support.

The Problem

There are certain things that, with an enterprise infrastructure, you need to count on. The one that I've got to call into question right now is timeliness.

Software development is a process, and it takes time. When software is updated, it gets released by the authors, and at that point, it is a candidate for inclusion in a distribution. Only the really bleeding-edge distributions include software as soon as it's released by the authors, though. Most wait a period of time for bug reports to come in and fixes to be released. Because RHEL is a commercial distribution with an emphasis on stability, it doesn't usually include new software releases as updates at all. Instead, it typically only accepts newer minor versions of the already-stable software that the particular version of the distribution shipped with. And each of those are tested for potential problems.

CentOS, by its very nature, is always playing catch-up. This isn't really a big problem, in and of itself, because in an enterprise infrastructure, it's not a good idea to apply updates willy-nilly anyway (even if they HAVE been tested by the authors, bug testers, users, and distribution publishers). If you test updates in a lab environment, then roll out updates according to a schedule, your odds of encountering a show-stopping error are heavily decreased.

Unfortunately, there are certain conditions where having up-to-date software isn't just a nicety, but is required. Some of those conditions are when the software you want to use is dependent upon a version of software that your distribution provider hasn't released, or when a security hole is discovered and is patched by the author of the software in question.

In those cases, CentOS puts us in a bit of a hard place, due to the very nature of their product. It's a valid argument that if you absolutely need security updates, then paying for commercial support makes sense, but there are a lot of us in the "leave it" side of "lump it or leave it" argument - mostly because our companies CAN'T afford to pay the support costs and still be profitable.

For that reason, we are at the mercy of our distribution provider to be timely with the software releases, and rather than spend money that we don't have to pay for software updates, we have to be particular about the sources of software that we install on our machines.

I do appreciate CentOS and all of the people who put work into it. They are doing a service for the community and should be lauded for their efforts. That being said, as system administrators, we need to be pragmatic and make decisions with the best interests of our infrastructures in mind, not our ideals.

Here's a graph showing the time delay (in days) from when Red Hat releases a version upgrade to when the CentOS release happens:

Clearly, there has recently been a staggering delay in regards to the 6.0 release, but even excluding that, the trend is clearly a rising length in delays. I'm not involved with the CentOS project, so I don't know what's causing this. There are a lot of factors, I'm sure, not to mention that Red Hat has not made it easier on them, despite assurances that CentOS wasn't being targeted by those actions.

On the mostly-non-technical side of things, there have also been some wrinkles in CentOS. At one point in time, the leading team member went MIA. Granted, things eventually settled down, and to their credit, there hasn't been any news of that sort since then (at least, that I'm aware of).

When I look at the 10,000ft view, I do have misgivings. I wish there was a greater transparency into the process. I wish I knew if they needed more resources (and if so, which ones). I guess it's frustrating to me because I don't know, and I feel like I need to know.

At the very least, when any product displays issues like CentOS has had, you owe it to yourself and your users to examine the alternatives. Fortunately, there are some.

Other Options

The leading competitor to CentOS is Scientific Linux, a recompilation of RHEL put together by CERN, Fermilab, and other labs around the world. I have not experimented with it much yet, but I have the ISO and I'm only waiting on enough uninterrupted time to get a feel for it. Before you play with it, you might take a look at the Scientific Linux Customizations that they've made.

I checked the delay in days that SL has been subject to, and compared them to CentOS, and got the following graph:

As you can see, both have had delays as of late. Again, I wish I knew if this were just the inherent complexity in the task (which has to be nontrivial), or if there were other problems.

There are others, as well. The wiki page RHEL Derivatives lists a good number, but I've got no experience (and have heard next to nothing about many).

In the end, we have three options. We can pay for RHEL, or we can switch to something and hope it proves itself stable, or we can stay with CentOS and hope it stays stable. I don't know what the right answer is, honestly. If I could see into the future, I'd be doing something much more lucrative than system administration, though! If I can leave you with one word of advice, it would be to construct your infrastructure such that the underlying operating system is as unimportant as possible. It won't be possible to completely insulate yourself, of course, because even with the best configuration management solution, you still need to update patched software, but at least with an abstracted infrastructure, you can migrate away from a particular vendor if or when they prove themselves to be unreliable.

What is your take on this? Where do you see the future of free Enterprise Linux? Are you making contingencies? If so, please let us know!

Edit

You may also find the following article interesting reading: "The Rise and Fall of CentOS". I hate to ruin the ending, but the ship sinks and Dag Wieers resigns from the devel mailing list (which I did not hear about until just now), though his repo still supports CentOS, so at least those of us with machines using it won't be left out in the cold.

  • http://mymcp.blogspot.com/ Steve Lippert

    Matt,
    I ran into the same problem a month or two ago when I heard that RHEL was already on 6.1 and there was no hope in sight for CentOS 6.0. That being said for the time being we've decided to stay with CentOS and even stay with 5, at least until our Peer1 production servers move up to RHEL 6. Given that there is no direct upgrade path between 5 and 6 I'm not really looking forward to the re-installs as I don't have anything like Puppet or Chef for my infrastructure.

    In defence of CentOS they choose to push 5.6 out ahead of 6 knowing it was going to push 6 back farther, after asking their user base. Also, I'm guessing Scientific Linux has more resources to throw at the problem than CentOS does.

    Like you, I wish there was more I could do to help the process along, but I'm not sure what they need. The best I can offer at the time is to help seed the distro on Bittorrent. I also run my own internal repo for updates to help save their mirrors some bandwidth.

  • Scott

    This is essentially why I've focused on Ubuntu for my distro of choice. It has it's own issues that it brings with it, but I'm allowed (and encouraged) to run the exact same code on my supported servers as I do my unsupported ones.

    And then somewhere along the path I got bit hard by that Debian Way thing and now I chafe at RedHat-style config layouts :-)

  • http://furicle.blogspot.com/ furicle

    Not having run RedHat since 7.x days, I just wonder - if you really really need that new software - can't you just break down and package it yourself? I didn't think it was that difficult to roll your own RPMs, and version them in such a way new 'genuine' packages would superceed them once they're released....

  • Kevin

    @furicle: You could roll your own packages; it's only a .spec file that specifies what files are needed, a few commands, and then you have your RPM.

    Now you have to test it for linking against all other software it might interact with, and test it to make sure it doesn't crash, and then test it to make sure it does what it says it does.

    Now what happens when the new software that you are running gets updated? Repeat the whole process again. Now multiply this by several different pieces of software and you wind up with a need of about 60 hours per day.

  • David Magda

    Another option is to only run RHEL in some places, and CentOS/etc. in others.

    So for your public facing infrastructure (web, DNS, mail) RHEL may be a good idea because of the fast, direct security updates. But for things like internal wikis, and DEV/QA machines, a bit of lag won't be as big of a deal.

    This is what we did at $(WORK-1): our production Oracle RAC systems used RHEL, but the DEV stuff CentOS VMs. Perforce servers used RHEL, but internal wikis and most product build machines used CentOS.

    The thing about CentOS is that it tries to be completely binary compatible with RHEL; Sci Linux does not. This may be important or it may not.

  • http://www.paulgraydon.co.uk Twirrim

    Increasingly I find myself of the same opinion regarding CentOS. We're going to start looking at the upgrade path soon and I keep finding myself wondering whether the appropriate option is to replace rather than upgrade. Weighing against it is we're already running two different distributions, one of which we're already commiting to moving away from. Looking after 3 during transition could prove to be a pain in the arse.
    Some of the security bugs that have slipped through Ubuntu disturb me a bit, and it's a little too bleeding edge for comfort in a production environment. I've been burned that way before (one of the sysadmins at Claranet when this happened). Enough that whilst I'm happy running Ubuntu on my desktop I'm not comfortable with it being production.

    The lack of dialogue from CentOS is a little strange, given how vocal other distributions can be. It's rare for a month to go past without there being a call for people to help triage and aid patching efforts by Ubuntu, for example. CentOS completely fails to take advantage of an enviably wide user base through it's poor communication. Their website increasingly makes the project look dead. If you skip over the CentOS 6 announcement and go trawling through the rest of the content you mostly have news articles a couple of years old+. Really? CentOS has been involved in nothing news worthy in over 2 years?

    furicle: Yes you can break down and package newer software yourself. It's not the age of the software that worries me in production, it's security patches first and foremost, with bug fixes trailing in not far behind. For a while CentOS was falling noticeably behind RedHat on their bug patching too, which really is dodgy territory. Patching is a key part of well rounded security practices. As a sysadmin I need to be as confident as I can be that if someone somehow breaks into my environment that my servers aren't going to be easy to break into either.

  • Steve

    Why be tied to an RPM based distro at all? Why not consider Debian or Ubuntu? Also, why be tied to Linux at all? Why not give Solaris or FreeBSD a try?

  • Alice Wonder

    While the CentOS 6 release was a little late, as were some of the 5.x security patches, talk of jumping ship to another clone is I think a little premature.

    TUV/CentOS isn't fedora, the life span of previous release is not ridiculously short. I'm not going to be migrating any of my 5.x systems to 6.x unless I'm doing a clean install on fresh hardware, and even then, I may stick with 5.x.

    Upgrading for the sake of upgrading is only necessary for the short lived distros, I use Ubuntu for my home desktop and there it is both necessary and nice to install the latest greatest. For servers though, a lot of stuff can break when upgrading so it really only should be done when the release is nearing EOL or there is something in new release you really genuinely need.

    Thus, for a lot of us, CentOS 6 isn't late because we would still be running 4.x or 5.x even if CentOS had released 6 the same day as TUV.

    I really prefer the philosophy behind CentOS of bit for bit binary compatibility, and that is something the other clones don't put as much emphasis on.

  • http://commandlion.com Joseph Kern

    It seems like the choice between RHEL, CentOS, and Scientific Linux isn't easy. They all seem to have their own issues. Have you tried Debian Stable? I would hesitate to consider anything else (except Ubuntu Server LTS (which is very close to Debian)). I can feel your pain with CentOS not having common packages (rrdtool took me part of a day to track down packages and install).

  • http://www.ewanleith.com/blog Ewan

    I think a lot of sys admins have had to make this choice recently, personally I switched to ubuntu lts, primarily because the other RHEL clones aren't available on the majority of web hosts.

    It's not a trivial change, but functionally ubuntu is put course very comparable to centos

  • Rhys

    I support a developer heavy shop, so that lag in software updates/features caused me far more trouble than CentOS was worth. Depending on how long I expect a system to be running, I either put debian stable or Ubuntu LTS on it. The more downtime I can have (if I have HA for example) the more I lean toward Ubuntu.

    Canonical has let some very big things slip by, which really makes me not trust them, but I think the tradeoff for so many people supporting it is worth it. Example: they changed their server kernel images to tickless somewhere around 10.04.1. That was 2 weeks worth of debugging our asterisk cluster, which would pause absolutely everything for ~5 minutes. Syslog and all other logging just showed a 5 minute gap of -nothing-. Then there was that other LTS bug where if you rebooted with a LVM snapshot it would fail to boot. Nasty, but I get far more fixes than bugs.

  • fpee

    ubuntu is quite often not an option for enterprise hardware, oh it works for some, but it doesn't work for a lot. I've used many different nas/san gear over the years, and you have to run rhel/centos or suse. You going to hack the driver to get it to work under a different OS, then run your mission critical app/database on it? I don't think so (that being said on a mission critical server it should be pretty easy to convince management to spend the $ for a rhel/suse license).

    The problem with Centos is, that it is a very small development team, and they have been essentially turning away newcomers. If you search the centos-devel mailing lists for the last 8 months you will find many people looking to help, and most getting turned away, or left with no help.

    I submitted a handful of (minor) patches to centos 6 in january, they were all ignored for months. really doesn't help my motivation to help the project.

    I ended up using scientific linux on a few servers as the performance gain from centos 5.6 is rather massive for what I'm doing. However at my organization we will be converting to centos6, as it is more binary compatible with rhel, and people are already posting problems with Xen on scientific linux that don't exist with centos 6.

    anyways. now that the files that needed to be modified in centos6 have been modified, it is quite a bit easier to roll out a new version, so I don't think that centos 6.1 will be that long of a wait. They are also now going to do a rolling release (like scientific) and going to be more public about how to help the project (so they say).

  • Andrew

    So you've plotted delays against releases, and said that the reasons for wanting things more up to date are software dependencies, and security patches. For dependencies that can't wait, compiling your own is always an option, and for security updates, I'd be more convinced by any evidence of in-the-wild exploits that the previous version was vulnerable to but the newer RH was not.

    That said, I am sure there are places where RHEL is the answer, places where Scientific is the best choice, and places where Debian is the best.

    As it is, I'm glad Centos 6 is out. As you point out, 'environment abstraction' makes this more possible than it used to be.

  • Alice Wonder

    @Andrew - For security patches, even where wild exploits exist, you can see the CVE list for RHEL and if it worries you, apply patches to the src.rpm yourself (using mock preferably) using a release tag that allows the "official" update to replace yours. Thus the delay in patches that sometimes happens really isn't that big of a deal.

    Most of the security issues though do not have remote vulnerabilities if any wild exploits actually even exist. I think I've packaged my own patched RPMs for CentOS only twice since 5.0 was released.

  • Howard

    I personally feel that the fee paid for RHEL is a good way of supporting a company that supports more kernel development than any other company (12.4%, as of the latest report from the Linux foundation, at http://www.linuxfoundation.org/docs/lf_linux_kernel_development_2010.pdf).
    As a developer, I also appreciate the incredible stability of the platform for the projects I work on. There are of course times that I wish I could deploy to a more frequently updated platform, but on balance Red Hat does a very good job backporting major features into their updates, and I have yet to find an instance where I have been limited by the age of the software in RHEL.
    tl;dr
    I understand the ideal of free software, but the actual contribution numbers by independent developers do not seem sufficient to move linux as a whole forward.

  • cos

    re: "ubuntu is quite often not an option for enterprise hardware"

    I'm old enough to remember when Linux simply wasn't an option for enterprise hardware. These things change, y'know.

  • d.f.

    We do a bit of a mix of RHEL and CentOS at our shop. Granted, the CentOS stuff we do is fairly trivial and confined (for the most part) to some utility machines that administrative staff need, rather than developers.

    Initially, I don't know as I'd be as concerned about lagging on the releases behind RHEL. I wonder if that is being burned by so many .0 versions of various RedHat releases that just. plain. sucked.

    I guess I haven't looked into this, but I'd assume a .0 release of CentOS would have some amount of bugfixes already rolled in.

  • Alex

    The amount of delays with each CentOS is getting ridiculous. I understand that it's free software, and a bunch of developers working on it for free - but a surprising amount of people/companies are now dependant on CentOS. The Cathedral model is simply failing with the CentOS updates.

    We're living in the days of easy distributed development thanks to git, github, and various other development improvements over the years. I'm surprised that nobody has had the motivation to either create a credible fork/alternative to CentOS, or put more effort into improving an existing alternative such as Scientific Linux (which doesn't seem to be quite as strict as CentOS and suffers from problems because of it).

  • Aaron

    I can't find the exact article I was reading in regards to this topic, but check out: http://lwn.net/Articles/430098/

    The gist of it is, if I remember correctly, Red Hat was upset\afraid\angry about Oracle re-branding their distribution (unbreakable linux) to try and turn a profit. Because of this, with RH 6 Redhat did things such as release kernel source without all of the RH specific patches to make it harder for Oracle to redistribute their distribution. The unfortunate thing is that open source projects such as CentOS get hurt in the process. While I have no idea how much of an impact this has had on CentOS, I blame Oracle for this whole fiasco :)

  • James Jelinek

    Great article, Matt.

    Our shop is starting to move towards Linux for a lot of front-facing servers and also determining what we Microsoft products we can replace. I've been a fan of RH for quite some time and at one time did go commercial. CentOS really seems to be a good alternative for those who don't need to be bleeding edge and have a somewhat stable/well-planned environment.

    That being said, I think I'll deploy a VM or two for some services and see how it runs. Definitely motivated me to get moving.

  • http://betteradmin.com Brian

    @Aaron: It has been said many times by CentOS developers themselves that the change you are referring to has absolutely no impact on CentOS whatsoever. It is meant to hurt companies who offer *support*, and as such require the ability to single out patches that were made to the kernel. CentOS does not offer support, they just need to apply the lump patch and compile it.

  • Marti

    I bought a refurb PC and installed CentOS as a desktop. The GNOME is comforting, but I know that underneath it will be harder to run.

    My main PC is an Ubuntu LTS but I am simply not looking forward to their Unity interface. (Hopefully, the LXDE I recently installed will not be broken by the 12.04 LTS.)

    I wonder how the lag times between RHEL and CentOS/SL compare to RHEL and Oracle and the PUIAS? PUIAS from Princeton indicate they are older than CentOS.

  • Pingback: Какво се случва с CentOS? « JustNick's Corner

  • Pingback: CentOS 6 Angers Me Again | Standalone Sysadmin

  • LinuxAdmin1

    I really like CentOS, they have 6.2 release out now and that is pretty speedy compared to the time they had to get to 6.0 release.

    I only hope all of these distro's continue on and the developers do not get burned out.

    I give kudos to all who do that type of work, as it is very challenging.

  • http://betteradmin.com Brian

    @LinuxAdmin1: The main problem still remains. The project runners refuse to open the process to the public despite many pleas of those who want to help. One of the project leads, KB, has dipped his toe in the water and asked for a little help here and there, but pretty much everyone else seems vehemently opposed to running the project in any sort of open manner. The excuses used for keeping it closed are easily defeated by even the most basic logic, despite the fact that the project itself is completely based on other open source projects that easily address whatever perceived "problems" there are with open source.

    I'm grateful for the work that's been put into CentOS, and I use it and will continue to use it every day, but I really wish the project was more focused on creating a real open source community.

  • Matthew

    At work all the mission critical servers run RHEL for the support factor. Items that don't need support as much like development boxes and a few workstations ... including mine) are relegated to CentOS (currently 5.6). Frankly all my scripts work flawlessly between the two and I get reliable and predictable results. THAT is the point of CentOS for us. Having the "Latest and Greatest" and the rat race that goes with it seems to be a fools errand. Reliability and stability are king in our house. Both RHEL and CentOS provide these. At home I use whatever I want, but my main box is now CentOS 6.2 ... openSuse 12.1 on my laptop and I think my current "toy" machine has a 1/2 broken Gentoo install ... I don't discriminate, but ....... at work its all about providing the best value to the business.

  • Sam

    Now that Centos 6.2 is out, I will probably stick with it... however in the meantime I was trying out PUIAS, which I found to be a worthy alternative, from http://puias.math.ias.edu/

  • JustUser

    No one even mentioned to pay a few bucks to support even not RedHat company but programmers around the world to compensate their work. Shame on you all, regular userrs!

  • http://www.standalone-sysadmin.com Matt Simmons

    JustUser: I appreciate the work that the RedHat teams do. But $300 a year (or $1200 a year for fun things like clustering or virtualization) per server for support is a bit...extreme.

    If I wanted to build a highly-available cluster using cman, I can see paying for support. It costs money to develop the software and maintain it. But a minimum of $2,400 for a 2-headed server? Ridiculous!

    I had 60-odd servers to manage when I wrote this post. If I licensed all of them with the bare-minimum $300/year, that's $18,000 out of my budget (a budget that was considerably less than $18,000!).

    In order for me to run my infrastructure, I need a free Linux distro. Since it makes sense to use a uniform distribution, and I like the technical side of RHEL, CentOS (or another RHEL clone) makes the most sense.

  • http://betteradmin.com Brian

    @JustUser: If you are referring to making donations to the CentOS project, they have been disabled for years, so there's no way to donate money to them. They do, however, appreciate participation in documentation and support efforts.

    Bit otherwise it is a good point, yes, one should try to support the upstream if they can, and also all the open source projects that feed into it.

  • Mike

    Why not use freebsd? Stable, timely releases and good support. I prefer linux for desktop but freebsd for server cant be beat for stability.

  • http://9999 dr.f9

    Have you tested Debian GNU/kfreeBSD?
    imho, not bad at the biginning. But it's interesting to see it as a domain controller server...

  • oshanz

    Charts need to update.

  • Pingback: 6 Things You Should Know About CentOS Linux - rackAID LLC

  • Pingback: CentOS joins forces with RedHat | Standalone Sysadmin

  • http://swwfm.net/ Robyn

    Hmm is anyone else having problems with the pictures on this blog loading?

    I'm trying to determine if its a problem on my end or if it's the blog.

    Any suggestions would be greatly appreciated.