Tag Archives: documentation

A Whole New World

I just got home from my first day as a sysadmin at Northeastern, and I think I’m really going to like it.

Since this is my first position in academia, I can’t generalize and say “wow, academia is so much different than private enterprise” (I’ll leave that for others who know better than I do), but I can tell you that it’s going to be a really interesting period of acclimation.

The infrastructure that I’m dealing with is widely varying. The College of Computer and Information Sciences has been around since 1982, and I suspect there are still small lingering parts of that infrastructure lurking about. While part of the infrastructure is utterly modern (particularly the user-facing portions such as the managed Windows and Linux desktops), there are certain portions that are visibly datable to the early 90s.

The “onboarding” process (does anyone else hate that term?) reminds me of how we used to bring people on at IA when I started there. It’s pretty rough and it could definitely be streamlined and automated, but when there’s an average turnover rate of around seven years, you have to wonder if it’s really worth the effort.

What is worth the effort is unifying the knowledge base. There IS network documentation, but it’s pretty well spread out, questionably updated (my predecessor actually left a year ago), and inconsistent. Creating a cohesive documentation repository (one which can, hopefully, be updated in an automated fashion) will go a long way to making the infrastructure faster to pick up for whoever has to do this next time.

Also up for renovation is the monitoring system, but I’ll save that for a later episode of This Old Infrastructure.

3 Activities That Will Make You a Better SysAdmin

As a number of my blog entries will attest, I really like drawing inspiration from sources outside of the typical system administration skill-set. I think that the technical aspects of our job are far easier than the “soft” stuff, or at least the “big picture” things.

When we start as inexperienced admins, we focus mostly on technical work. Ideally, we perform this work at the behest of our more experienced superiors, or less ideally, because there’s no one else to do it, and the company’s owner is telling you that the mail server is broken and that it needs fixed yesterday.
'Server Rack' photo (c) 2008, Jamison Judd - license: http://creativecommons.org/licenses/by/2.0/
I still recall very well back when I was a first year administrator and Brett, our senior admin, would have me do some small bit of technical work. Usually it was to build a new service or rebuild an old one. I vividly remember wondering to myself why he was having me do these kinds of things. There was no outward indication that something was wrong, I hadn’t heard anyone suggest these new services. It just seemed like Brett was randomly having me do things.

It took some time before I got to the point where I could really see the forest for the trees. Brett wasn’t being reactive, he was being proactive. He could see impending problems and build solutions before they became emergencies. It took a long time of dealing with individual components of the network, then small subsets of services, then the entire infrastructure before I could see why he did what he did.

The valuable addition to my skill set wasn’t technical in nature. It was the ability to see the relationships of systems to each other. It was gained through perspective. I was looking at trees when I was building solutions, but Brett was looking at the forest when he’d direct me to do them.

Your goal as a System Administrator should be to get to the point where you see the forest. It’s not an overnight kind of thing, and it takes a lot of working on trees before you can see it, but you should be working toward that goal while you’re doing the grunt work.

I’ve thought about it, and I’ve identified three things that I think can help you to move into this architectual-type role. These aren’t the only things you should be doing (by far!), but if you do these three things, you will advance your analytic skills above and beyond someone who doesn’t.

Also, they might seem a little out of left field, but trust me, the skills that you gain from them will carry over into your eventual role as someone who deals with forests instead of trees.

  1. Read Airline Accident Reports

  2. This sounds macabre, but what these reports consist of is ex post facto review of incidents involving aircraft by people who are extremely well trained and who have a very large incentive to determine the various causes of the incidents.

    The best resources in the world are probably from the FAA. There are two major data sets available, the preliminary data, which is a brief report that covers the apparent facts and situations (but may be subject to change through the course of the investigation), and the final reports, which are the results of what is sometimes a multi-year investigation.

    To give you some idea of the thoroughness behind these reports, here is an almost 100 page document detailing the crash of a 13 passenger turboprop airplane. From the abstract:

    The safety issues discussed in the report address fuel system limitations, requirements for fuel filler placards, and guidance on fuel system icing prevention. Safety recommendations concerning these issues are addressed to the Federal Aviation Administration (FAA) and the European Aviation Safety Agency. Previous safety recommendations concerning crash protection for airplane occupants and flight recorder systems were addressed to the FAA.

    These people are not screwing around.

    One thing to keep in mind while reading these incident reports is that, for the purposes of fact finding, some investigators will assign a single root cause (which, given the redundancy of modern aviation, is typically assigned to “pilot error”), while others cite contributory causes and situations. An incident investigator named Sidney Dekker wrote a gamechanging book about this called The Field Guide to Understanding Human Error, where he discusses that simplistically assigning blame isn’t just wrong, it’s actually counter productive, because the problems don’t get fixed. I picked up the paperback ($20 used), and I loved it.

  3. Read Medical Case Reports

  4. Speaking of macabre…I think it’s important to read medical case reports. Like I said, it sounds like it’s from left field, but it’s actually very relevant.

    The best site that I’ve found for this is The Journal of Medical Case Reports. It’s a publication dedicated to just that: case reports. They’re available for free, and they’re all written by practicing doctors who have documented interesting or unusual cases that they’ve encountered that they feel would be useful to their fellow medical professionals.

    Here’s an example: Low back pain during pregnancy caused by a
    sacral stress fracture: a case report
    (pdf). The medical jargon isn’t as important to us as the form and the function. Check out this abstract:

    Introduction: Sacral stress fractures are a rare but well known cause of low back pain. This type of fracture has also been observed as a postpartum complication. To date, no cases of intrapartum sacral stress fractures have been described in the literature.
    Case presentation: We report the case of a 26-year-old Caucasian European primigravid patient (30 weeks and two days of gestation) who presented to our outpatient clinic with severe low back pain that had started after a downhill walk 14 days previously. She had no history of trauma. A magnetic resonance imaging scan revealed a non-displaced stress fracture of the right lateral mass of her sacrum. Following her decision to opt for nonoperative treatment, our patient received an epidural catheter for pain control. The remaining course of her pregnancy was uneventful and our patient gave birth to a healthy child by normal vaginal delivery.
    Conclusions: We conclude that a sacral stress fracture must be considered as a possible cause of low back pain during pregnancy.

    Read through the rest of the paper. The frank and open discussion about the case, the evaluation, all of it lends itself as an evaluation by a knowledgeable professional giving his opinions to his peers.

    The doctors who authored this piece are doing exactly what we should be doing. How many times have you encountered some strange condition or a case that seemed bizarre, but when you discovered the underlying cause, was preventable through easy checking early in the process, if only you had thought to do that?

  5. Write Reports of Your Own
  6. You had to see this coming.

    Now that you have two very excellent examples of reports generated by professionals in their respective fields, you should come to see that you need to start writing reports on the things you experience, too.

    Part of being a professional is taking part in the professional community and sharing your knowledge with your fellow practitioners. The strange things that you notice can end up being very helpful to other system administrators. By publishing your findings, you are legitimizing your experiences and your professional status, plus documenting your experiences for posterity, plus encouraging others to do the same. There is absolutely no downside.

    So where to publish? Every conference that I know of is actively seeking practice and experience reports, which would cover either of the above types of reports. If you had a failure that you ran a post-mortem on, or you encountered an issue during the otherwise routine operation of your infrastructure that others could learn from, then by publishing a practice and experience report, you help everyone.

    There is also a journal for system administrators called ;login:, and they are always looking for case studies, which is what this kind of report would be.

Doing these things will not turn you into an amazing system administrator overnight, but taken together, they’ll provide excellent models of professional behavior that we should all work to emulate.

You don’t need to do any of these things to be good at your job, and I don’t want to give that impression. What I would like you to consider is that, given two equally proficient administrators, I will always prefer the one who has tried to share what he or she has learned rather than kept it to themselves.

OpenOffice trick to auto-increment IP addresses

I’ve been fleshing out my IP address spreadsheets, and I’ve been using a trick that I didn’t know about for a long time, till someone on a forum showed me. It was indispensable.

You know how, if you want a series of numbers down a row, you can click the lower right hand corner, and drag it down, and it autoincrements the numbers for you? Well, in oocalc, if you try that with IP addresses, it just copies the original IP address. I don’t know why, but it does. MS Office gets it right, heck I think even K Office will do it. OpenOffice, not so much.

The trick to doing it is to put a space in front of the IP address, first.
”″ will allow you to drag it down as far as you want, incrementing the last octet.