Re: System Administration needs more PhDs

Tom Limoncelli has a post up today entitled System administration needs more PhDs.

He makes some great observations and brings up a lot of interesting questions. The one that I think the others flow from is “Why are good practices so rarely adopted?”

My opinion, gained through observation, is that sysadmins arise from one of two places. Either they start out in relative isolation, or they come from an environment with multiple systems administrators.

The former develop their own ways of doing things through trial and error and/or research. This leads to endless ways of accomplishing the same or similar tasks. The utter heterogeneity of possible platform combinations lends itself to having each admin reinvent the wheel.

The latter typically have an established infrastructure in place, a well defined set of hardware, and a much more rigid structure of procedures and usually a bona fide methodology for change management.

The reason that the standalone sysadmin almost never resembles the well trained sysadmin is because best practices all seem to be vendor driven, reliant on a subset of devices and situations, and are hidden as well as possible behind support and contract agreements.

Those are hurdles the lone sysadmin faces AFTER he has discovered the “optimal solution”, whatever that is. You mention puppet. Should you use cfengine or puppet? Unless you know about puppet, you’ll use cfengine, unless you haven’t heard of that either, in which case you’ll roll your own. In my experience, you’ll find $betterSolution right as you’re implementing $bestSolutionYouKnowAbout.

I don’t know whether there are more sysadmins in a single environment than in a plurality, but there are a _lot_ of sysadmins out there by themselves.

By themselves, sysadmins rely on their own cleverness, but together you get a synergy of ideas. The whole becomes smarter than the sum of the individuals, but most sysadmins never get to experience that. That’s one of the reasons I started my blog. To shed light on what other people are doing, how they operate in their organizations, and so on.

Your books are a great resource for sysadmins, but the lone sysadmins of the world need to start communicating between themselves, and with the “institutional” admins out there. The same solution won’t always work, but the sharing will go a way toward a meritocracy of

  • James

    “Why are good practices so rarely adopted?”

    Because good funding for equipment and training is so rarely allocated.

  • Reamer77

    I think some people have made attempts, but I’m not sure why they don’t catch on. I was a member of USENIX/SAGE when a previous employer paid for my membership. I honestly think the price is steep, and didn’t renew it on my own because I didn’t feel I got much out of it. LOPSA seems interesting too, but they don’t seem to have gotten off the ground. Apparently, they are in a lawsuit with USENIX (I had no idea). I’ve also tried going to Linux User Groups meetings, but they never seemed to have anything really interesting for sysadmins.

    Besides expensive conferences (ex. LISA), there doesn’t seem to be anything that fills the void of sysadmin collaboration. I would think that someone could fill the void quickly.

    Or maybe sysadmins are lazy and loners by nature :)

  • Matt


    Yea, and even if you GET the money, you’ve still got to decide which equipment to get and training to attend…


    I think the price is steep too. It doesn’t seem like there’s any representation for the sysadmins who are on their own. They assume that an organization has an IT department. I wish!

    There’s probably some truth to the sysadmins being loners, but I don’t think it has to be like that. This blog is proof that sysadmins will speak up and contribute ideas. I get hundreds of comments a month from people who want to share information. Everyone who reads this site is interested in learning about how other people do things. 5000 people a month come here, and I think that’s a pretty sizable segment of people, considering the niche we’re in. It’s a sign that the institutional sysadmin isn’t the only one out there.

  • Craig March

    Spot. On.

    From my limited experience the problem stems from another two areas as well.

    Firstly, IT deployments in smaller companies grow organically according to current demand. For instance I have seen that the implementation of a new database required a complete overhaul of the underlying operating platform. Fair enough, but the overhaul was rushed in time for the new database to go live. The end result is a consultant delivered system with limited time to train those who will administer it. The likelyhood is that is going to remain the situation for the next 3 – 5 years.

    Secondly, good admins (and some bad) will always want to leave for bigger and better challenges. (Isn’t that the kick we all get from it? I don’t want to be explaining to managers how they can read their emails or how to copy and paste files for much more!). With staff turnover you waste so much time waiting for new staff to fully understand the IT system they now manage. Documentation helps but lets face it, has anyone ended up in a job where the documentation was even just adequate, let alone good?

    Oh the life of a standalone sysadmin!

  • Nick Anderson

    Reamer77, I also used to be a member of USENIX/Sage and also of LOPSA. I got very little benefit from either. In fact the only reason I decided to join LOPSA was because at the time they had a deal where you got a free subscription to sysadmin mag which went defunct about 2 months after I joined, and I never saw a single mag.
    I work at a small/mid sized company and the budget for IT is so small. I get creative in ways to make 5 bucks act like 500. DSL and Cable modems in the office where we do EVERYTHING online. There is no room in the budget for a reliable link.
    Trying to get configuration management going so that some day if/when I do leave for bigger better things the next admin will have less to worry about. Although puppet is the best thing around (at least that I know about) its horrid. Seriously requiring a webserver to do configuration management? Right on their site they say the internal webrick server wont scale past 50-100 nodes without beefing up hardware (beefing up from something with 2Ghz and 2GB memory). Im sorry but a machine with 2GB memory and a 2Ghz processor should be able to do configuration management for several hundred machines without issue. Maybe I am an idealist and over zealous with efficiency in software. Then again maybe its just the ruby and smacking taste of rails that taints me on puppet.
    I do agree with Luke Kains that our tools suck. I wish I were more of a developer, but frankly I don’t have time with everything that I take care of to even try to become a developer.

  • Anonymous

    One other comment that folks didn’t touch on here is time. It seems like a running gag that sysadmins live at work, never take vacation, and eventually go insnae due to overwork.

    I really have to believe that only part of the time issue is bad time management on the part of the actual admin. Unfortunately, the time demands placed on us are often unrealistic, I’m sure we all have horror stories about that.

    Still, its not like administration is the last bastion of bizarre practices…if that were true, the archives over at would be empty.