The Coming Future of System Administration

Date March 6, 2012

Four years ago, I wrote a blog entry titled The Future of System Administration. It started out with "Here’s a hint…it’s not sitting at a shell prompt typing out commands."

In it, I discussed how system administration would become development, much more like coding than like swapping disks...

The future is already here — it's just not very evenly distributed.

--William Gibson

Basically, yes, it has come to pass. No, not everyone is using that particular strategy, but the number of people who aren't will continue to get fewer and fewer, especially as cloud-based offerings get more and more prevalent, accepted, and expected. But all of this is just tangential to what I really want to talk about.

System Administration has to divide in order to continue to grow. Tom Limoncelli has been saying it for a few years, but I did't really get it until relatively recently. I don't know if we agree about the underlying reasons, but I suspect we're on the same page. Let me lay the ground work, and then I think you'll also agree that it's pretty much inevitable.

System Administration is a relatively recent field compared to pretty much every other professional endeavor. Medicine has been around in an organized form for over 100 years. Accounting for 400-500 years, and considerably longer. (Note: this is not to say that the art and science of these are not much, much older, but I am speaking here of organized professions).

The profession of System Administration is still coalescing, but that's only because we're still very much in the formative years of the art and science. It's exciting, because we get to experience the transition from humors and leeches into germ theory and beyond.

As we've progressed, something has become clear; a single person is unable to handle all aspects of the job. We've all seen this, of course. There are administrators specific to networks, storage, virtualization, applications, and more. This division has been caused by the domain-specific knowledge required to adequately perform a certain aspect of running IT. Not everyone can know everything...

It is this last point that is going to cause the division that I speak of. Currently, the field is very roughly divided into junior and senior sysadmins. The Job Descriptions for System Administrators short booklet by USENIX is a good effort to add some qualification to the distinctions between the various job levels, but it still supposes a continual advancement from the very entry level administrator up through senior, level IV System Administrator. I don't think this can continue for much longer, particularly in an infrastructure of any complexity.

As time progresses, here's what I think will need to happen (and in some cases, it's already happening, but it's going to become more widespread).

We're going to start seeing two career paths for people who work with servers. At the top of the professional career paths, there will be the people who are in charge of physical infrastructures, and there will be people who are in charge of the operational infrastructures. The inevitable move to cloud-like infrastructures is going to mean that, to the operational architect, the physical hardware is almost completely irrelevant. At the same time, the physical administrator is going to be concerned with those aspects of running the infrastructure, such as physical Moves, Adds, and Changes (MACs) as hardware becomes more numerous and anonymous.

Almost no one is the size of Google, so when machines die, they will need to be replaced. They will need to be monitored, and their performance, independent of the software installed on them, will need to be monitored for efficiency, with underperforming equipment being fixed. The cloud layer of abstraction will allow machines to be pulled in and out of service to be repaired and replaced, but certainly a degree of expertise will required to engineer the best, most efficient datacenter solutions. This is what the top tier physical infrastructure will concern themselves with.

The other side of the coin is the operational administrator, who concerns themselves primarily with ensuring the business operations and continuity. If the physical infrastructure admin's concerns are hardware, the operational administrator's concerns are data and software. The operational admin won't ever have to consider what disk array to purchase, or how large the storage needs to be, but will instead concern themselves with things like the performance growth rate and the number of CPU units required for a particular task. Infrastructures will be developed like software is now, using the same methodologies, sharing some of the tools, if not the languages. Someone in a senior operational role will need business intelligence, as well, because they will be responsible for translating business directives into infrastructure.

These specialized roles will have junior members, of course. An entry level physical infrastructure admin will probably enter the field in the same way that many of us did, as an apprentice, then moving to a journeyman type of level while learning more. I imagine that more education will be necessary to fill the truly senior physical roles, but it's not inconceivable to learn everything on the job.

In the operational role, it's harder to imagine many people making the same transition from non-admin to admin. My own path, from Tech Support to SysAdmin, won't be nearly as common. Many current admins were able to make the transition between non-admin and admin because the operating environments are similar on their laptops and their production infrastructure. Once the operating system ceases to be important, this easy path will disappear, and the only people who make that transition are those who actively work at it.

Because the operational aspects have such an emphasis on programming, it's easy to see that more administrators will become sysadmins directly out of CS programs at universities. Hopefully the people who dictate the curriculum will take notice of this and start adding operational electives.

I don't have a time frame for this, but I suspect that in another four years, we'll see more and more companies coming online with this model. Over the past couple of years, I've seen a lot of blog entries and online articles talking about the disappearance of System Administrators. These articles are mostly hyperbole. The sysadmin of today is not going to disappear, but they are going to be fewer in number.

Companies who, right now, have their own physical infrastructure are not going to go through a lot of expense to get rid of it, but companies who come online in the future are going to see more and more reason to use a public cloud service, which means that they'll only be employing the operational administrators. Sure, there will still be desktop support in most cases, but in terms of sysadmins, for those companies, it's mostly going to be architects. Once a company grows to the point where it's not economical to host at Amazon, they may build their own cloud infrastructure, in which case they'll employ the physical infrastructure admins, too.

Just so you know, this isn't a dystopian future. Specialization is a good thing. It's going to require us all to work together, but the end result will be that we can accomplish much more than we could before. The knowledge level required to do our jobs is going to go up, but we should see this as a challenge to learn more and to be better.

  • Jack Hughes

    A recent battle with Microsoft Exchange certainly back up your specialisation point. If I wanted to get a decent grasp of how to do things in Microsoft Exchange that'd be pretty well all I'd have to do. Ditto with an awful lot of other apps out there. They are getting so big you just can't keep many in your head at a time.

  • mckain

    Well, yeah, the actual sysadmins will disappear from the small/mid companies, but what about the huge high tech companies like Amazon services, Google, Yahoo, FB, I guess they will concentrate all the SysAdmins like you said divided in networking tasks, storage, virtualization, applications, and more.

  • Scott

    I think you've got it right. Sysadmins aren't going to go away but our roles will change. Much like they've always changed. I started in this business stringing 10Base2 cable to 8086 boxes running an IPX stack. There's little-to-nothing from that time frame that still applies to the things we do today. And even 20 years ago there were specializations. HP-UX sysadmins couldn't necessarily find their way around a Windows LAN or a Cisco Catalyst router.

    Part of the reason that I love my job is that what I'm doing today probably won't be the same thing I'm doing 5-10 years down the road. And yes, we'll have to specialize. I ditched the exchange infrastructure at our 30-person company because it made more sense to outsource that management (We chose Google, though you could easily go with the Microsoft product if you were so inclined)

  • Aleksey Tsalolikhin

    I'm glad you're looking at this, Matt.

    I would name these two divisions as "system administrator" and "system engineer". It's the system engineer that is more concerned with data flows and the overall big picture viewpoint, how does this system interact with other systems, how is this system used, how does it interface with the business, etc.

    I do still see the overall career path as going from system administrator to system engineer. The administrator has a narrower scope, and doesn't have to know quite as much as the engineer.

  • Steve

    I think the more accurate terms are System Administrator and System Architect. System Architects will retain value within a company - they translate business requirements and processes to the required architecture. System administrators, on the other hand, will become more of a commodity as IT infrastructure itself becomes more of a utility-style commodity over coming decades.

    It's not a bold claim to make at this point that System Administration jobs will become increasingly outsourced as this occurs. This goes hand-in-hand with this blog post though. As System Administrator jobs migrate away from enterprises towards specialized outsourced utility companies, the role of System Administrator will also become much more specialized.

  • Aleksey Tsalolikhin

    Good point, Steve.

    The trouble I have with both terms (engineer and architect) is, in general English, this word means a certified and licensed professional. We lack such certification and qualification of our computer engineers and architects, which is troubling as modern civilization increasingly runs on computers and depends on soundness of computer systems.

  • Pingback: Furture of System Administration | Kenyan Patriot

  • Lobo

    I definitively agree with this analysis, since I work in a company that is already there (at least partly). We specialize in running client servers (virtual and physical), and almost exclusively work with clusters or other High Availability solutions. So basically we are the "System Administrators" while the customers handle the "System Architect" role. Normally we have no idea what kind of applications are running on the servers and it's very rare that we are even involved in the decisions on what applications to use. Usually the customers just say that they need their CRM-system running 24/7 and together we'll calculate what kind of storage, servers and so on that is needed and then "My" company will make shure that the servers are up and running, that any SQL clusters functions correctly and so on. The client handles the application, decides when there should be downtime or when to upgrade.

    In reality we'll need quite a bit of cooperation and sometimes there is a start-up period before everything runs smoothly, but the businessmodel is very successful. Our clients are quite often larger companies with their own IT-department but they still outsources businesscritical systems to us, since we are experts in just these types of systems. We've been doing this for quite a few years and are not the only ones in Sweden doing it. I believe that we will see more and more of these companies and probably even companies that utilizes the different cloudservices, and then selling this rather than the client signing up to amazon, microsoft or google themself.

  • Lawrence

    I used to be the System Administrator that did small, then medium companies. Then, I wanted to be the System Administrator that could do everything in large enterprise.

    But, things are changing as we outsource things that make sense, and things that didn't make sense. And, the cloud is certainly changing how things work. Though so far its between...people wanting us to provide simply cloud like services of infinite capacity and people signing up to cloud services to circumvent IT and policies (such as IT security.) Or, being able to do things that don't make sense, like letting developers develop on production systems....if they mess it up, its easier to recreate the production server in the cloud.

    When I started here, I was seen as someone that would best fit as an operational system admin....because a 90lb weakling is going to have trouble racking 1U servers, let alone a 4U one. (plus I came from an enterprise software development background - where there was a division between "software developer" and "software engineer"....the "software engineer" also being a "system administrator" and knew/understood the system side of making software work well....the big picture. -- I was hired as "software developer", but after a reorg...the division was left with only one IT person, and he only did Windows. So, I became the Unix the end, I was doing 100% Unix admin...)

    Though being short handed here...I am doing almost everything....but when I started we all had areas of primary responsibility and areas of secondary (backup) responsibility....(initially, all secondary, but over time more and more primary.) I get to swap lots of disks now, because its like I'm the only one that knows how to do that from previous experience.... I'm the only one here where this hasn't been my only IT exposure.

  • emag

    What surprises me is that, even in my "smallish" organization, which has traditionally been solely HPC-based, I'm seeing the same distinctions. What used to be the "Sysadmin" group is now divided into "Storage" and "Servers and virtualization". At the rate we're going, I can see the latter group slowly dividing into 2 groups. This is only in the 4.5 years I've been here. There was always a separate "networking" group that we work closely with.

    As it is, the S&V group (of which I've recently become the team lead) is pretty much already progressing down this path. There's simply too much to know, and too much to do, for any one person to fully be involved in both worlds. We're still trying to find a workable method of dividing work, to the point where some of the "storage" people are still the official admins of various (mostly legacy) systems, but it's clear that the writing is on the wall.

    On top of that, we're also looking for "System Architect" positions, though I feel we're erroneously trying to meld all three groups, storage, servers, and virtualization, into a single position. It's been clear from out difficulties in hiring someone with expertise in all 3 areas that we really should break it out into separate roles, if only we could afford it (and find the candidates... this is supposed to be an up-market for employers, right?!).

    I'm actually looking forward to the transition. I know the areas that I like, and I often revert to writing code to perform parts of my job. Some of my coworkers are more base-hardware oriented, and can talk to the pros & cons of choosing one storage subsystem over another. And still others are much more "propeller-heads" that spend almost all their time mired in code, helping our scientists optimize things to produce results faster. Not grouping us all under the same generic umbrella will probably help us all, in the long run.

  • Pingback: Catchup bookmark blog 201207 » J's blog

  • Pingback: Behold Microsoft, Barbinger of the Future | Standalone Sysadmin

  • mrG

    I'm agreeing with the gist of this, and frankly had already in my own mind largely diverged the hardware sysadmin, which I am not really much of myself, from the O/S & Network sysadmin that is concerned, yes, with a form of software, although I'd say that form is very different from the sort of sysadmin that handles applications configurations (which I also do) and different again from the straight applications coders (a field that more and more I don't recognize anymore).

    But what I also see in my encounters with vendors and clients is a preponderance of sites where there is no sysadmin at all, perhaps one programmer thrust into the job and doing it badly, and to that end I'm surprised it isn't easier to find federations of sysadmins of all the various types bound into a sort of consortium consultancy that can do the day to day running of servers (the way most box-hosting colos will do) as well as the late-night emergency hardware or O/S crises troubleshooting.

    Do you think there is too little trust in companies enough to share root passwords with outsourcers (and perhaps not even knowing it yourself) or could the global confederacy of freelancers pulling together for-the-moment ad-hoc workgroups be a sort of sysadmin IT outsource of the future?