The Future of System Administration

Date August 19, 2009

Here's a hint...it's not sitting at a shell prompt typing out commands.

I'm sitting here on hold, waiting for Dell support to come online and tell me that my laptop is out of warranty, so I've got some time to think, and I'm reflecting on some of the sysadmins I've met lately, as well as some of the technologies I've seen for enabling administration, and it all seems pointed in one direction. Sysadmins are becoming developers.

If you think about it, in a sense, we've always been developers...developers of systems, that is. Infrastructuresmiths, you might even call us. In my mind, though, there's always been a delineation between the admins who run the systems and the programmers who program them...at least there has been since the early sages who wrote the software and the operating system. Maybe that's just my own inexperience talking (if so, please, comment and let me know!).

Administration is definitely getting away from the method that I "grew up" learning, which was "fix it right the first time so you don't screw it up, then document it when you get the chance", where "fix it" meant log in to the machine (as root, of course), edit the configuration by hand, test the syntax (if the service supports that sort of thing), then restart the service and check to see if the edit worked. If what you "fixed" broke something else, then you'd scramble to fix it, and restart the service again. Lather, rinse, repeat.

It's probably an understatement to say that there are better ways of going about this.

For a while, best practices flirted with development by storing machine configurations in version control repositories like subversion. You would make a change to the config, commit the file to the subversion repository (along with notes regarding whatever you did) ,and if your change broke the service, you just rolled back the change. It was a big improvement, and similar in methodology to how developers manage source code. This was really the first step for us.

Now, a few years later, we're being introduced to another paradigm of system administration: framework-based configuration management AKA cfengine, puppet, and chef.

These configuration management packages offer exciting new time-saving abilities. Not only do you get configuration management, you get it centralized and programatized. I can't say I have any experience with cfengine or chef, but what little experience I have with puppet has taught me that it's more programming language than configuration file, and best practices are to treat it as such. Store it in a subversion repository to keep it safe, and to roll back changes when needed. You can even use subversion to create development testing branches and the like.

If we're not becoming developers, we're blurring the line quite a bit. This isn't necessarily a bad thing, but it does require us to update our skillset a bit. CS degrees which teach proper development techniques are going to become more valuable. Those of us without those particular skills may want to keep them in mind for the next time we get some free time to work on improving skills.

Michael Halligan said the other day on twitter, "I feel that any sysadmin today who isn't learning Ruby and either Chef or Puppet will be unemployed in 5 years".

That might be overstating it a bit, but knowing up-to-date system management solutions will never be a hindrance, in my opinion.

Are you using anything remotely modern on your systems, or do you still use the "change and scramble" method? And don't be afraid to share, we won't judge. Chances are really good that we've all seen (and done) worse!

  • anonymouscoward

    meh, unfortunatelly i still do it "old style"
    i'm thinking for some time to change a lot in how i do things but for some time something else comes up that needs to be done "right now!"
    hope i'll start learning at least a bit this year..

  • http://www.standalone-sysadmin.com Matt Simmons

    @Anonymous

    I think a lot of us do. I've only recently started dabbling in puppet. I don't even have my configs in subversion. I'm way behind.

    It seems like there are "levels" or "tiers" of administration. And by that, I don't mean that it has to do with the skill or competency of the administrators, but of the degree of sophistication by which the machines themselves are administered. It's something we all need to work on and improve, but only once we know what "better" is, can we get there.

  • http://blog.ryankovar.co.uk/ bread555

    I literally had this same conversation yesterday. I work with 5 other sysadmins, but I do the primary large scale Configuration Management work (thus the Nagios work) and scripting. As I go on, I find I am having to use things like SVN or SCM, IDEs, and developer development methodology(Agile/RAD).

    Also, as I use more and more opensource software, I find myself having to do things like "Edit C" and "rewrite PHP". Not necessarily the tasks that I was told I would be doing when I started System Administration.

    Perhaps the reason for this blur between System Administration and Developing is that as our Enterprise systems become more stable (Compare Windows NT 4.0 and 2003) and need less and less daily hand holding, the value of employing Senior System Admins is not their ability to resolve issues everyday but rather their ability to reduce the amount of issues from occurring with extensive configuration management. Since every environment is different, this includes a lot of hackery.

    And as far as me... I can see the city of ConfigManagement on the hill, but my visio map has not yet led me to nirvana.

  • http://twitter.com/digitalbigsky TinaC

    Amen brother!!!
    My problem is convincing "Damagement" that it matters. Afterall - they won't be here at 2 AM fixing things.

  • John Allspaw

    The point you make is absolutely the truth, but I'll add that install, config, and even release management is only part of what sysadmins do, of course, at least in the field of web operations. Those areas, while hugely important and clearly going thru a great step in evolution due to the new tools being built, allow for the *real* hard stuff to be worked on with focus.

    Systems architecture work (should we shard? Memcached on every box or a dedicated cluster? Cost per user calculations? Self-healing mechanisms? Load feedback to the app from lower layers? Etc) is the work that, while certainly are helped by the current and excellent trend of automated infrastructure, isn't something that can be done with code, as in: 'fix it and forget it'.

    I absolutely agree with your main point but will argue that these new paradigms only serve to rid us of the boring work so we can get to the hard work. :)

  • http://www.standalone-sysadmin.com Matt Simmons

    @John

    You make a great point. Does anyone REALLY want to ssh into 50 (or 500) machines to edit sudoers? Not a chance.

  • Twirrim

    Puppet is one of those things I really want to get to use in practice but haven't had the time so far. I've read plenty of literature on it and the excellent book "Pulling Strings with Puppet", but a complete lack of time at work and a crazy busy time at home with house moves and wedding planning has left me little time to experiment with it!

    Now here's the big question for me from the article:

    Why Ruby? I don't mean to come across as a perl-zealot, I'm rather programming language agnostic, but what does Ruby really offer me as a sysadmin over Perl? I can guarantee Perl will be on pretty much any *nix based system I might ever get my mitts on. Ruby is no where near as likely.

  • http://blog.ryankovar.co.uk/ bread555

    @John, I agree. Our tools are getting better are doing the dummy work, so we can work on the fun stuff.

  • natxo asenjo

    For all of you that have not yet invested time in a configmanagement solution, I urge you to seriously look at cfengine. I know, I know, all the kewl kids are using puppet now, but cfengine kicks serious butt.

    Both cfengine and puppet are descriptive "languages". Both have plus points and minus points, but cfengine is the most proven solution with thousands of deployments in big server parks. If you choose puppet, that's great, but do not just choose one without knowing why ;)

    With cfengine (if you do not have a windows domain) you can also administer your windows infrastructure. This comes in handy, it works. You can manipulate the registry or copy files. If you do not want/need a windows domain, this will let you administer everything with the same interface.

    I am deploying cfengine2 (I know, the old one, who cares, it works) at work and man am I glad I started.

    As to why ruby? well, if you install puppet you need ruby. You do not need to know ruby. I agree with Twirrim about wondering why ruby, but hey, this is a free world.

  • http://www.standalone-sysadmin.com Matt Simmons

    @natxo

    I've heard from a lot of people that cfengine is great. I haven't seen a direct head-to-head, feature-wise comparison between it and puppet, though. I do know it's popular with a lot of people, though.

    As for "why ruby", I don't know, as I don't write it, but people tell me that development is incredibly easy with it. I've got it on my list of things to check out, though.

    Thanks for the comment!

  • xyz

    @twirrim: Its not as much as what Ruby offers to you as a sysadmin, its more of what Ruby is offering to the developers of the sysadmin's configuration management tools, like Puppet and Chef. I'm guessing its because Ruby is generally considered suitable for making DSL (Domain Specific Language) applications.

  • xyz

    BTW, the video of Open Source Bridge's Configuration Management Panel is published:

    http://osbridge.blip.tv/file/2278426/

    It features a discussion between the developers of AutomateIt, Puppet, Cfengine, Chef and bcfg2.

  • http://www.standalone-sysadmin.com Matt Simmons

    @xyz - Great link. Thanks a lot!

  • http://blog.opscode.com Adam Jacob

    I just wanted to give a +1 to what the inimitable John Allspaw said. As one of the authors of Chef, and a career Systems Administrator, the thing that really changes when you start using these tools is that all the stuff you never really liked doing anyway just disappears. The places where you as a truly professional systems administrator have always added the most value (architecture, troubleshooting, tuning, capacity planning, etc. etc.) become the focus. Getting a great automated infrastructure running means the System Administrator gets the "right" thing streamlined enough that a developer can just do it - but you still need that expertise.

    As for why Ruby and not Perl, it's a question that is close to my heart. I love Perl, and it was my first choice for writing Chef. But Ruby as a language makes it incredibly easy to make very natural extensions to the language - Modern Perl can come close (with things like Moose and Devel::Declare) but it's not quite as natural. One of our goals with Chef is to enable truly polyglot automation - you should be able to write your recipes in any language you want, and simply use Chef as a library. We have a working prototype of writing them in Perl up already at GitHub and CPAN. An example of what I'm talking about:


    # The ruby version
    package "sudo" do
    action :install
    end

    # Make sure sudo is always at the latest version
    resource package => "sudo", sub {
    my $r = shift;
    $r->action("upgrade");
    }

    The top is Ruby, the bottom is Perl.

    I'm happy to answer any questions about Chef, or automation in general. I'm adam-at-opscode-dot-com. You can also find a whole posse of us on irc.freenode.net in the #chef channel.

  • http://www.standalone-sysadmin.com Matt Simmons

    @Adam

    Thanks for dropping by and for the comment. It's truly a different world where we're headed as sysadmins. It's going to be fun!

  • http://www.cmdln.org/ Nick Anderson

    I've been dabbling in configuration management for a couple years now, but only recently was I finally able to deploy a complete system. It is not a far reaching system, but it seems like the perfect fit. I looked at cfengine, bcfg2, and puppet; I ended up choosing puppet. I hadn't heard about chef until recently. I've taken a cursory look at it and it seems to be a fine tool. In my situation I still think puppet or maybe bcfg2 (if i could get over xml making my eyes bleed) is the best fit. I have several other admins in my group and I work with developers and QA engineers. Not all of them know or have any interest in learning any full languages. It's my plan to take configuration management to all of these groups and allow each group to manage their own configurations. So far everyone seems to be excited about it and the first deployment using a configuration management system has gone very well. Its definately a skill that I will continue to hone. I think working with configuration management systems also makes you think about how to structure your infrastructure which is never a bad thing.

    I don't think that we as sysadmins are becoming developers though. More recently it seems we have borrowed from some developer tool sets because they suit our needs well. I have always been of the mindset that scripting knowledge (in multiple languages) is a requirement of any sysadmin worth his salt. Not that you need to be an expert by any means, but we deal with text and often repetitive text structures and repetitive tasks. The only way to efficiently deal with data like that is write scripts to automate your processes.

    What I am waiting for is developers to start learning from us! The biggest issue I have *always* had when working with developers is developers want the new shinys over almost everything else. I love the new shinys as much as the next person but my primary goal is stability and reproducibility. Lots of hand rolled packages and bleeding edge technology is typically not a great companion to stability and reproducibility.

    I am still a fan of new technology. For example Xen is one of my passions. I don't run the latest xen kernels in production but I do think Xen is superior to Vmware so I run what I feel is the superior technology (even though its kind of new and shiny).

    @Matt
    Time to get that subversion server setup!

    I am just completing a new subversion server at work. It hooks into corp ldap for authentication and I have a nice little script that will adjust the commit author to the real name of the person instead of the svn username (using ldap its our employee id and quite ugly not to mention unhelpful when looking at a log). I may do a writeup once I get it complete which should be soon. Let me know if you need any help getting subversion up and working (its pretty easy). Of course you could use git its much nicer imho but if you do use subversion you can still use git-svn on the client side and kind of get the best of both worlds.

  • http://www.cmdln.org/ Nick Anderson

    ehh gad, I didn't realize the comment was so long, I was writing it in between diaper changes and the like.

  • http://www.cmdln.org/ Nick Anderson

    “I feel that any sysadmin today who isn’t learning Ruby and either Chef or Puppet will be unemployed in 5 years”.

    I would go out on a limb here and say he is quite wrong. First of all I would say its a mistake to pin down any one technology like Ruby for example. Did I miss something or did python, perl, shell (all staples of system administration IMHO) become defunct? What about the other configuration management systems? cfengine, bcfg2, automateit are all great projects. And spacewalk sure looks interesting as well. So beyond making such a specific prediction I think that there will always be admins who do not script much or use configuration management systems. Everyone is always learning and new people come into the field all of the time (they can't know it all when they start). But I think the sentiment is solid. You will be a better sysadmin if you practice and extend your skill-set. Knowledge of scripting/programing and configuration management certainly count in my book as skills.

  • http://toastedsandwi.ch Legooolas

    I've been using Puppet for a while, and chose it over cfengine because of the higher-level nature of it which looked more pleasing to me. I've not looked at Chef, Capistrano, BCFG2 etc much at all, since I wasn't going to switch until I found something terribly wrong that I couldn't do in Puppet -- It would be much like switching languages for all of your scripts at once, unless you ran two configuration management systems side-by-side during a transition. Anyway, it sounds painful :P

    If you have a new project which you can spend a small amount of extra time on the setup for it, then I'd highly recommend using one of the config management tools and using SVN or some other version control to keep the config history. I've not even particularly had to roll back anything much, as I have separate branches for dev->staging->live for config as well as code.

    “I feel that any sysadmin today who isn’t learning Ruby and either Chef or Puppet will be unemployed in 5 years”.

    Higher-level sysadmin work is definitely the way of the future, and I can understand the core of this statement of "we need higher-level tools or we're screwed". However, I fully expect that in 5 years time there will be a plethora of other tools which will replace the words Ruby, Chef and Puppet :)

  • http://ryannedeff.com Ryan Nedeff

    Unemployed because you don't know Ruby? Maybe in some organizations, but not as a whole. I can't imagine trying to pin down the admin world with one language.

    I think the statement is true if you rephrase the statment as "Any Systems Administrator without passable knowledge of one or two languages and some sort of Configuration Management Suite will be unemployed in the near future."

    I think any one or a mix of Perl, Ruby, C, .Net, PHP or any other array of languages is something every sysadmin should at least have a grasp on. After all, the more tools you have in your tool box, the more likely you are to be able to overcome an obsticle.

    Limiting yourself to one tool is bad news. If all you have is a bucket of kerosene, the more likely you are to think you can rip out the driveway easier by burning it up. But, then again, we come back to great depth vs breadth debate, which I think is beyond the scope of this conversation. :)

  • http://www.cmdln.org/ Nick Anderson

    Well lets have the debate !

  • http://bashinglinux.wordpress.com/ Dan Fruehauf

    The usage of a source control system is inevitable for SysAdmins today. Before being introduced to puppet, I had my handful collection of configuration packages, all made up of idempotent scripts (in a source control of course).
    Puppet obviously makes my life easier.
    I can identify with saying that SysAdmins are becoming developers. We're looking for a new SysAdmin now at our workplace, looking at the CVs we've got up until now we're talking about apes in disguise of SysAdmins. I'm looking for a developer that would fill up my place of maintaining tens of thousands of lines of scripts, will be able to catch up with puppet quickly and be able to respond to complex demands from developers or customers.
    Wearing both hats of C++ developer and SysAdmin I can admit that the work I'm doing on either is pretty similar, the common things are:
    1. I write code in both
    2. I use a source control in both
    3. I have to think before acting
    The differences:
    1. One language is compiled the other is interpreted
    2. Once it runs as root and once it runs as a regular user
    3. Usually in scripts I do not have to take care of memory leaks as they run only for a very short period :)

    @Matt
    Point being is that I totally agree, if you are a good SysAdmin, you are actually a developer.

    Besides, a question for all:
    A product would usually contain not just compiled code, but also a handful of tightly coupled stuff that is also a part of an application, such as packaging the product, configuring the OS for the product, some support scripts for the product, etc. All of these are responsibilities of who? a SysAdmin in a development team? a developer? a SysAdeveloper?

  • Matt

    At my current employer, nobody wants to change anything, ever. Rather than begin implementing something useful like cfengine or puppet, the masses here would prefer a broken/cobbled together mishmash of scripts smarter people wrote 10+ years ago. Over coming these barriers makes it a real challenge to keep your skills up to date, let alone develop them further.

  • http://www.standalone-sysadmin.com Matt Simmons

    @Matt

    That does make it difficult. Especially when there's already a "solution" that people trust. And honestly, if the solution works, maybe it's not so bad.

    But if you ever find yourself in the position of re-engineering the equipment, maybe you should make it a point to gradually edge in modern configuration management.

    Good luck. I'd be interested in hearing more of your infrastructure (not to mention the trials you probably go through to get stuff to work)

  • Twirrim

    @matt and @matt

    Matt S's approach is an approach I've participated in in the past. Slowly but surely brought in new systems under the guise of 'updating and improving', which was a secondary gain next to getting a single consolidated and manageable way of doing things.

  • rb

    I'm still using the old method, but I'm currently managing 1 production-ish server and 1 private server. as soon as get my new, quadcore VPS machine though, I'm switching to puppet + krb/ldap. after a year of manual management, my "production" machine is beyond a mess.

  • http://www.linkedin.com/in/williamtriest William Triest

    Ultimately, I think the two really are similar in that both take a problem and find a technological solution to meet the business needs; one just uses more off the shelf components than the other. The core troubleshooting skill set remains the same and hopefully the security awareness of both is the same. Testing is slightly different but has a strong overlap. I don't think jumping between sys. admin. and programming is that much further then jumping between different programming platforms (depending on the platforms).

    I should admit, I am VERY biased. My current position has been struggling with the fact that technically I am now a DBA but I know things their systems administrators don't (previously I had shared responsibility for windows servers), I have a deep understanding on networking, I know some about security (I'm usually consulted on SQL injection issues), and I've successfully completed programming projects (primarily when its a non-.NET language which our official core competency). We're currently in the process of a reorganization and no one is sure which pillar to put me in, since I do work in all three (my philosophy is they pay me, so while I'm there I do whatever is in their best interest assuming its legal and moral). I'm known as some one who can pick things up very quickly, and I honestly credit that to the fact that I have technological interests across the board and have been fortunate enough to be exposed to talented individuals in the various aspects of computers.

  • Jean-Francois LEHE


    “I feel that any sysadmin today who isn’t learning Ruby and either Chef or Puppet will be unemployed in 5 years”.

    For me the truth is "I feel that any sysadmin today who isn’t learning ITIL will be unemployed in 5 years”.

    So the tool (cfengine, puppet, ...), is not the greatest part of the configuration management.

  • Jean-Francois LEHE

    This topic is about configuration management, i thnk rather than choosing which tool or language learning, why not simply consider learn ITIL ?

  • http://www.cmdln.org/ Nick Anderson

    For those who want to know more about ITIL.
    http://www.itlibrary.org/

  • http://livingonthecloud.blogspot.com/ JR

    Interesting blog Matt.

    I'm not persuaded by the Ruby statement but the general premise of the article is interesting. I've found Infrastructures.org quite an interesting site that provides some starting points for system administrators to infrastructure administrators.

  • Risar

    @ Jean-Francois -- because ITIL doesnt solve the technical problem associated with running large scale IT with ever-shrinking budgets. (I'm ITIL certified - trust me on that)

    Consider aviation, automobiles, warfare, plumbing .... basically any other industry or technology. They didnt build the stealth bomber before they flew at kitty hawk. They didnt have water reclamation and desalination plants before they had basic plumbing.

    IT is still a technical problem and will be for a long time - treating it as marketing, sales or some business centric division just creates a large competitive advantage for the other guy. Your people will spend months firefighting, filling out tickets, forms and performing data entry never getting to address the core problem - which is still a technical issue.

    ITIL is perfect for non-technical IT shops, highly SLA centric IT shops and similar situations where the work being done should be minimal change and rigidly organized. Rapid change, engineering environments and reactive/adaptive environments either crash and burn (all productive work stops) or immediately trim the ITIL process down to a loose framework of "ITSM best practices" that are automated, but still make the business happy. (eg the svn/cvs/git repository is "release management" and similar concessions)

  • Darin Lory

    WOW!! I'm glad I'm not the only one thinking I was doing more "developing" than sysadmin functions lately. I've always done some software configurations and web server setups in the past, and in the distant past, I used to do actual software development in COBOL, Fortran, C/C++, Jyacc JAM, Informix 4GL, Sybase, and NeWS (Display PostScript).

    In the last couple of years, I'm doing more of compiling and fixing software that we buy or setting more open sources applications that "developers" used to setup. The developers of the past are now the "project managers" of projects, and usually have the development work outsourced or pushed to a UNIX sysadmin.

    Lately I've been setting up Apache servers with Catalina and Derby and fixing some of the code that some Indian Developers are sending.

  • http://www.standalone-sysadmin.com Matt Simmons

    @Darin

    Thanks for the comment, and yeah, you're not alone. It does sound like you've got a leg up on the non-developer sysadmins, though, and that's great! Have you started implementing any of the configuration management solutions?

  • http://conigliaro.org/ mike conigliaro

    For those asking why Puppet was written in Ruby:

    "From Luke Kanies, Puppet's author: I was a sysadmin by trade and had mostly developed in perl, but when I tried to write the prototype I had in mind, I couldn't get the class relationships I wanted in perl. I tried Python, because this was around 2003 and Python was the next new thing and everyone was saying how great it is, but I just can't seem to write in Python at all. A friend had said he'd heard Ruby was cool, so I gave it a try, and in four hours I went from never having seen a line of it to having a working prototype. I haven't looked back since then, and haven't regretted the choice."

    http://reductivelabs.com/trac/puppet/wiki/FrequentlyAskedQuestions#why-is-puppet-written-in-ruby

  • http://luqmanux.blogspot.com/ Luqman

    I use Cfengine, and had to say, it's great for large infrastructure, however, with small ones I tend not to use it.

    Will try Puppet and Chef .

    Thanks for the nice post.

  • http://digg.com/users/rexsiteskrl/history MERLE CERVANTES

    I've been looking around http://www.standalone-sysadmin.com and actually am impressed by the great content material here. I work the nightshift at my job and it is so boring. I've been coming right here for the past couple nights and reading. I simply needed to let you know that I have been enjoying what I have seen and I look forward to reading more.

  • Pingback: Adam Moskowitz’s video: The Future of System Administration | Standalone Sysadmin()