Sysadmins aren’t (necessarily) programmers, they’re people who can program

I remember back when I was first learning Linux, and I encountered shell scripts. I wasn’t a programmer, and I didn’t “get” it. I distinctly remember thinking, “well, THAT’S something I won’t have to learn”. Ha!

As it turns out, I was incorrect. Writing scripts is an essential skill for a system administrator. In Linux/Unix, we’re blessed to have an amazing development environment, where as administrators running on Windows had to make due with batch files until the dot net revolution came along to introduce ASP and VB script. Now, with powershell, they’ve actually got a great environment to write systems scripts in. Between that and things like Windows Server Core, I’m beginning to wonder about the Redmond camp. But I’m digressing…

Writing scripts isn’t an optional tool in an effective administrator’s tool belt. It’s absolutely vital to efficiently performing many, many tasks. Personally, I use the bash shell, because it’s the default, and it’s what I started on. You should use whatever you feel comfortable with, whether it’s a shell script or perl. Heck, I’ve been desperate enough to even do a couple of things in CLI-mode PHP, just because I’m more fluent in it than I am perl (which is a shame, and something I’m going to be working on rectifying).

My last “fun” bit of shell programming was probably a cron job that checked for a new tsunami warning and emailed me the text of the alert. Before that, I wrote an entire RSS reader in bash. With bookmark support. Yeah, I’m a sick man.

What kind of fun things have you done with scripts lately?

  • I wrote a bit of python the other day to collect some information for an entire subnet of nodes as well as do some comparisons for some inventory purposes. The guy who was going to do the work planed on doing it manually for each node and expected it to take him an entire day.
    The same guy also had a need to put a bunch of ip addresses in numerical order. He did that manually also. I think it took him about 5 minutes. It took me 8 minutes to research and write a python script to do it. Yeah it took me longer, but I now have something I can reuse later if the need to sort ips ever comes up.
    Seems that I write some bit of scripting each day, even things like taking thread dumps of an application over a period of time was better served by a simple for loop in bash than me entering the command every once in a while manually.

  • =) I wrote the wordpress plugin that Matt is using to generate the “« Previous Entry
    What is Cat5e?” link

    My tricky bit of coding there was taking an array and finding the next earliest post and the next latest post. Which dosn’t sound that hard, and it really wasn’t, but getting the data from the undocumented function was a bit of a pain. Then for fun I had confused which order the data was in.

    Then I wrote my own custom CMS, it can even generate static html and use pretty urls. But boy is that code ugly, I don’t even think I want it connected to my name.

  • Learning Ruby changed my career as a sysadmin. I previously used shell – bash and some ksh. Perl never clicked for me. Ruby helped me understand not only how to write reusable tools, but it was the “A ha!” catalyst for understanding object oriented programming. Now it helps every day in writing Chef recipes to automate infrastructure.

  • Chris

    First, great blog, glad I found it. I’m firmly of the camp that if you’re going to have to do something more than twice on a system, write a script. My go-to is perl, mostly because it’s what I learned first 12 years ago, and it seldom lets me down. The corpus of modules available for it is amazing. In the last two weeks I’ve written relatively straightforward scripts to create LDAP users from our MS-SQL based student information system, and a script to convert data from our legacy library system to our replacement Koha system. Standing on the shoulders of the CPAN giants provides a phenomenal view :-)

  • My most amazing shell script, which I used for years to post and edit things on my website which, depending on the year, was mysql-backed php or perl/html::mason.

    Please enjoy.

  • natxo asenjo

    Perl is surprisingly good in Windows platforms. And much faster than Powershell (factor 3 faster is not unusual, and cheaper in cpu cycles as well). With the Win32 modules you have access to the whole windows api, so in fact you are then using native tools for Windows servers.

    If you have a mixed environment of linux/Windows knowing Perl will make your life easier. And if you follow the best practices for writing Perl code, it is no line noise. In fact, it is much nicer to read than the Powershell scripts :-)

  • In my last job, almost 5 months ago, a colleague and me had to write an information gathering system for every Unix in the platform. From a management server with remote access to every server, the job was launched weekly via cron and got useful info (CPU, RAM, server type, cluster version, hostname, IPs and many more).
    We did it in shell and the Unix OSs were AIX, Solaris 8 & 10, HP-UX 11i (the three versions) and Red Hat Enterprise Linux (RHEL 3,4 &5).
    The 2nd stage was to ask the EMC storage with Symcli and develop VB scripts for the WS2003 and get the info through samba from the core server.
    I have to admit I had so much fun, is unusual to have the opportunity to work in a such a wide variety of different systems, at least for me since I am almost exclusively focused in HP-UX and sometimes in Linux.

  • I once wrote a php front end that generated a bash script that configure linux machines as incoming mail relay proxies to novell groupwise servers (which at the time did not reject incoming emails for which it would not relay but bounced them after accepting them and so were constantly a target for spammers who thought they hit the motherload.) complete with more custom generated bash scripts generated by the php generated configuration script.

    That was pretty crazy…

  • Recently I wrote a bash script that will find all pdf files in the directory in which it is executed. Then it will start a while loop to convert it to pdf and in this while loop the txt version will be run through a command to find the occurrence of words (from a file) and output the word plus occurrence into a output file.

    Output looks like:

  • Ian

    I’ve started messing around in python. For simple scripts, I can run them on Windows or Linux, which is nice. I’ve only used it for a few things so far. Most of the tools I’m used to using for repetitive tasks are GUI tools. Most Novell related administration is done using one of their many disjointed gui interfaces.

  • I use the mutt mail client on one of my home servers. I missed the growl notifications that GUI mail apps enjoy on OS X, so I threw together a Perl script that checks imap from that server and sends growl notifications to my laptop.

  • I forgot about todo cycle python script to manage a GTD style todo list.

  • My first position where I was a quasi programmer, there was an “amazing” collection of bash scripts strung together that pulled jobs from a queue (read directory of files) and created some pretty complex answer files to drive CAD software. Users would run a bash script to submit their specifications to the queue. A couple of SGI IRIX workstations did nothing all day but process the queued jobs and cranck out customized CAD designs including GCode for the laser cutting and the sheet metal nesting layout. It was pretty neat to see in action considering some of it was never intended to be automated.

  • At one job I had to map out all the network connections in a cube farm. Network drops every so often, with four port face plates. After years of people moving, rearranging, etc. the poles were often (conveniently) behind filing cabinets :-( So i end up having to duck under a desk, or climb behind a cabinet, unplug the cable, and then climb back to the computer to see if i had disconnected the right one.

    I ended up writing a wrapper for ping that would loop through a ping and literally say the word “ping” (using rsynth, I believe). It saved me much time and aggravation.

    Since, I’ve started keeping a folder of little “throw away” scripts like that because you’ll never know what comes in handy later on!

    Another handy one is my “end of the day” script that I run on my laptop to do a mass check-in of all my projects in git and to backup my essential files to usb.

    The less painful you can make backups, the better :-)

  • Zach, great idea verbalizing ping.

    That reminds me of a script I wrote for an asterisk server once. The script took text, chunked it and posted it to, it took all the resulting audio clips and played them back to the user. I think the first application was reading weather to me over the phone. I wanted to use it for audible checking of cluster burns but never got around to consolidating the data. For anyone who is wondering why I didn’t use the available tts engines … have you ever heard the output from speex?

  • fishwalking

    Hah! All you NIX admins shall now tremble in fear as us Windows Admins harness Powershell to take over the world! All kidding aside, I’ve been learning and using Powershell to do everything from generate KPI reports every 5 minutes, to parsing Event logs, to even performing certain tasks when Nagios sends me an email alert. I can do coding, but it’s not something that I live for. Unfortunately it’s one of the necessary evils of the job, because I’m too lazy to push the buttons.

  • I totally agree that scripting and system automation is essential as a sysadmin. Once I realized how important scripting/programming was, I started learning Perl and the Bash Shell which turned out to be good decisions in the long run. Learning Perl and the Bash shell allowed me to automate tasks such as common file transfers over ssh and ftp to remote servers, and find documents with certain keywords.

    I’ve also played around with Windows Powershell 2.0 which has proved to be invaluable on Windows Servers. With powershell I have been able to write scripts that will e-mail me when a certain user is not logged in, check to make sure key services are enabled and if they are not start them, and gather system statistics and info quickly.

  • D.F.

    Oddly, I just had a conversation about this yesterday with one of our programmers. My best script was a real hackjob that would read a list of domain names out of some BIND config files, check to see if those domains had a whois record and if they didn’t delete the selected entries from the config files.
    Not to start a flame war or anything, but ever since I’ve seen the powershell stuff I’m reminded of an old maxim (damn I wish I could remember who said it) “Those who do not understand unix are doomed to re-implement it. Poorly.”

  • Anthony

    I find it interesting how many comments this entry triggered.

    I try to write scripts for anything that is repetitive or reiterative. I.e. things you have to do repeatedly, or a single task that has to do the same thing over and over.

    I generally use bash, though I don’t tend to do anything very complicated freqently enough that I remember how to do things very well – so I keep a collection of scripts I’ve written around that can use as a reference manual on how to do things.

    My perl scripting is .. ok – I’m not going to say how may years ‘stale’ it is. It’s very stale. But I also keep a collection of little perl scripts I’ve either written, come across, or had someone else write. I’ve found that as powerful as bash is, sometimes things are just faster/easier in perl. (Probably in Ruby and Python too but I’ve never had the need to learn yet another language)

    The most complicated bash script I ever wrote was a replacement for a 2 or 3 line script that controlled part of a batch process in a web application. They wanted suddenly to be able to completely customize a large number of configuration options that had just been added to the backend application, and it had to be managed without any changes to the web side of things, so they wanted users to be able to edit and modify the script to add their customizations.

    So I had to make a bash script that idiots could edit without breaking it. I think there were more comments and instructions in the file than actual code.

  • @Anthony: I find it interesting, too! I think I really struck a nerve!

    Everyone, your scripts sound amazing! Thanks for sharing them with the rest of us. And thanks, everyone for reading and taking the time to join the conversation!

  • I’m more a programmer than a sysadmin so if I have do something even once, I’ll script it. Tend to use Perl more than Bash because I’m more familiar with it. CPAN should be any Perl-loving sysadmin’s first port of call – you’ll save time and write cleaner, more robust code that you would just hacking something together yourself.

    In general, I think sysadmins should have know a full-scale scripting language that puts a good library at your fingertips. Doesn’t matter which – Perl, Python or Ruby are all good choices, PowerShell if you’re on Windows, Groovy if you’re a Java programmer doing a bit of sysadmin on the side.

  • Claire

    I write a lot of Perl, and over the years I’ve learned a lot more about programming than I would have expected. I want to plug Damian Conway’s Perl Best Practices, which not only gives you some really useful information about how to write clear and maintainable Perl code, but also helped me understand a bunch of programming paradigms that had never quite clicked for me.

    I also find myself using the O’Reilly Perl Cookbook a lot; if there’s something I’m not sure how to do, it’s probably in there, or the pieces I need to figure it out are.

  • Louis Munro

    I second (and triple) the recommandations for Perl Best Practices (and any books or modules by Conway really). It has made my perl (and progrmaming in general) much more maintainable and readable. I’m no longer embarassed to show it to strangers…
    I think sysadmins can learn a lot from programmers, especially about tools and methodology when it comes to scripting. Things like using a single editor and knowing it well, using a debugger, a revision control system, some ideas about modularity and reuse etc.
    Just the other I showed my colleagues how to create a Perl module to hold common regular expressions as objects.

    These days I’m working on my masterpiece: a set of Perl and Javascript scripts (using qooxdoo ) to parse Networker’s email notifications, stuff them into a relational database and interact through a web interface which you can use to query and annotate them for reporting (the default networker reporting is buggy and incomplete). Oh, and I want it to use PAM for authentication.
    It’ll be done by Christmas…

  • Scott Murphy

    Hi Matt;

    Lots of good items here. SysAdmins do real work too :)

    The most recent Q&D hack I did was a bash script to run around the compute farm and get the RAM, the number of physical CPUs and the number of cores for every machine (because nobody knew) and to generate from the site configurations the list of users who have access, the machines they can access, what queues they exist in and the list of machines with a list of allowed users, all because the vendor supplied tools do such a lousy job of reporting anything useful. I also had a whole hours warning of the meeting where this was being discussed.

    The former portion has been rolled into my machine deployment post install phase to update the motd with this information so all you have to do is log onto the machine to determine the info. The motd has the machine name, the OS release, the install date and the above information, as well as the amount of swap space (don’t ask why). No cfengine or puppet here yet, but I’m working on that. Perl and expect are my friends and I’m beginning to think I need to start learning ruby.

  • When I was a under-graduated I had some colleges that would like to be SysAdmins and they hated all programming lessons, in the same way all guys that would like to be developers hated all network classes.. That’s the reason why we got bad professionals at the business.
    Anyway, I’d like to see your crazy script to rss ;)

  • When I started as a sysadmin I kept coming across and having to interpret perl scripts. Thankfully even though they were awful I could generally figure out what the underpinning logic was of each section and figure out what impact that had on problems I was troubleshooting.

    I took a week off work, sat down with the learning perl book for three days and learned how to write perl scripts from scratch. On the fourth day I sat down and wrote a perl script to process some log files and e-mail me the results that I had previously been grepping by hand. From that point I was hooked.

    My motto:

    Do it twice, script it once.

    Obviously I’ll script it anyway if I figure out its going to be faster, but if I find I’m doing a task for a second time on a server, I generally figure it’s something I’m going to be repeating and look to see what I can do to automate it.

  • dminio

    Mine is not work-related. I have a Sansa Fuze, and the app to transfer video to it is simply awful. Not only that, it’s Windows-only and I prefer to run Fedora at work.

    I wrote a Perl “automatizer” for a third-party script that transcodes video for my Sansa Fuze. It takes all my video files in one folder (or grabs them from youtube by way of an other third-party script), transcodes them and moves them on my Fuze automatically (if it’s mounted). It does some temp file cleanup as well.

    It’s a hack, I know it, but it works well enough. I used to be a pretty good coder in university, but I don’t get much practice anymore.

  • rb

    I’ve been writing lots of little tool frontends in Django. endusers tend to not like the commandline, but if you give them a web interface, you get a good reaction. great way to manage LDAP user creation / password changes :D

  • I must agree, good sysadmins are not necessarily coders, but can program, and some can do it pretty well. Whether it be nagios check commands, cron jobs that clean wtmpx or SVM montoring scripts, I always find the act of “creation” very rewarding, and revel in the opportunity as I find creation isn’t always inherent in the job of sysadmin.

    One of my finest (and first) scripts I wrote a couple of years back was a bash script that used smbclient on a Solaris 2.6 machine to collect a list of machines on a Windows (98) network. That output was then parsed and used to mount each machine individually and backup the important files to tape using a combination of tar and ufsdump. The solaris machine had 4Gigs of disk space which meant there was no large storage area to temporarily hold the backups, so we were going straight from samba mount to tape. In the same script the local Solaris machine also had its turn in the backup schedule. This soluton worked successfully for a number of years before we eventually turned to using bacula on a linux box.

    Even better – restore worked too!

  • Carl

    My example is actually on the Windows platform. As any good lazy sysadmin (I’m a SAP basis consultant) I grew too lazy to login on every system to check availabilty, backups, certain login information (for SOX violations), etc. Basically daily checks. We have 8 odd customers with a crap load of servers that need to be checked. The task get spilt up between a bunch of juniors, but we need to check up on the lazy buggers that tend to slip up more than they do not. So I start automating it and mailing a simple report to myself. It started out as a simple DOS batch file that gathered informationto a text file and then mail it to me. I then added some database scripts to check backups and some other information (Was tricky since I’m not a DBA and had to figure some stuff out in 3 different database types. Oracle/MS SQL and MaxDB). Then I converted the whole output to HTML for cosmetic reasons for which I need some more intelligence than what was available in batch file so I used Kix32 scripting. Something I did years ago (10+)when I was a NT jocky/sysadmin. After that I wanted to do some nice graphical stuff for availabe diskspace and decided to take the plunge and teach myself vbscripting to do this. This is a an ongoing project of my own where I still tweak the whole process over time. I still want to rewrite the whole thing in one script language/environment and also rewrite it so that it is a single script that caters for all the different clients in rather than custom script for each one.

  • Sam

    I would have to agree that Chris and natxo that Perl seems to still be the common denominator after all of these years. And yes, it is very good on Windows too, especially if you make native(ish!) executables with PAR.

    My last decent bit of Perl was a Nagios plugin to query various aspects of SQL server performance metrics. I’m currently also writing an internal package manager (“pacman”) using a mixture of Perl on Solaris/Linux/Windows and some windows batch.

    I also created a nice Rails portal for internal use (“Lumberyard”) that provide our development team easy access to production log files without them having to have access to the production boxes annexed at various colos outside of our main network. I’m working up to talking my employers into open-sourcing it as I’m quite proud of it and think it’ll be of use to a few people. It’s quite nice that instead of batching periodic copies of logs to a central location it creates an SSH connection on-demand and sucks the log over to the user in real time, meaning they can simply refresh to get an updated log. Some AJAXified “tail” function would be nice. Maybe in the next internal release, or the first public one ;)

    I have to admit that I come from a different background, being a professional developer (in Assembler of all things) before moving into sysadmin and architecture work, so I perhaps have a different view of things. That’s the beauty of UNIX-like systems for me: they are made to be programmed at every level from the shell downwards, it just depends on how much abstraction from the kit you need to get your job down. It is a testament to the power of the UNIX userland and shells that quite often you don’t need to move beyond bash/ksh.