Non-SysAdmin Related: #MarioMarathon

I want to take a second for a non-sysadmin-related post…bcause I want to let you know about a good cause that’s happening right now.

The guys who run Penny Arcade started a charity a few years back called Child’s Play. The idea was that the gamers of the world get a bad rap. People like Jack Thompson spent their lives vilifying video games and the people who play them.

To set an example of gamers doing good, Jerry and Mike created Childs Play, which would take the money donated to it and buy video game systems, games, and other toys for children’s hospitals. Although it started relatively small, the current hospital map is really impressive.

Anyway, some friends of Ben “Funnel Fiasco” Cotton, wanted to do something to help Childs Play, so they organized Mario Marathon, where they would live broadcast video of them playing through every Mario Brothers game (or at least, all of them that existed at that time).

They’re now in their fifth year, and they have quite a sophisticated setup, a cast of maybe a dozen people that filter in and out throughout the weekend, costumes, contests, and you can watch them play in a quarter of the screen. It’s a blast, and it’s what is going to be on my TV all this weekend.

You should watch too. The video is live on Check it out and donate! Over 5 years, they’ve generated over a quarter of a million dollars for Childs Play. Amazing!

3 Activities That Will Make You a Better SysAdmin

As a number of my blog entries will attest, I really like drawing inspiration from sources outside of the typical system administration skill-set. I think that the technical aspects of our job are far easier than the “soft” stuff, or at least the “big picture” things.

When we start as inexperienced admins, we focus mostly on technical work. Ideally, we perform this work at the behest of our more experienced superiors, or less ideally, because there’s no one else to do it, and the company’s owner is telling you that the mail server is broken and that it needs fixed yesterday.
'Server Rack' photo (c) 2008, Jamison Judd - license:
I still recall very well back when I was a first year administrator and Brett, our senior admin, would have me do some small bit of technical work. Usually it was to build a new service or rebuild an old one. I vividly remember wondering to myself why he was having me do these kinds of things. There was no outward indication that something was wrong, I hadn’t heard anyone suggest these new services. It just seemed like Brett was randomly having me do things.

It took some time before I got to the point where I could really see the forest for the trees. Brett wasn’t being reactive, he was being proactive. He could see impending problems and build solutions before they became emergencies. It took a long time of dealing with individual components of the network, then small subsets of services, then the entire infrastructure before I could see why he did what he did.

The valuable addition to my skill set wasn’t technical in nature. It was the ability to see the relationships of systems to each other. It was gained through perspective. I was looking at trees when I was building solutions, but Brett was looking at the forest when he’d direct me to do them.

Your goal as a System Administrator should be to get to the point where you see the forest. It’s not an overnight kind of thing, and it takes a lot of working on trees before you can see it, but you should be working toward that goal while you’re doing the grunt work.

I’ve thought about it, and I’ve identified three things that I think can help you to move into this architectual-type role. These aren’t the only things you should be doing (by far!), but if you do these three things, you will advance your analytic skills above and beyond someone who doesn’t.

Also, they might seem a little out of left field, but trust me, the skills that you gain from them will carry over into your eventual role as someone who deals with forests instead of trees.

  1. Read Airline Accident Reports

  2. This sounds macabre, but what these reports consist of is ex post facto review of incidents involving aircraft by people who are extremely well trained and who have a very large incentive to determine the various causes of the incidents.

    The best resources in the world are probably from the FAA. There are two major data sets available, the preliminary data, which is a brief report that covers the apparent facts and situations (but may be subject to change through the course of the investigation), and the final reports, which are the results of what is sometimes a multi-year investigation.

    To give you some idea of the thoroughness behind these reports, here is an almost 100 page document detailing the crash of a 13 passenger turboprop airplane. From the abstract:

    The safety issues discussed in the report address fuel system limitations, requirements for fuel filler placards, and guidance on fuel system icing prevention. Safety recommendations concerning these issues are addressed to the Federal Aviation Administration (FAA) and the European Aviation Safety Agency. Previous safety recommendations concerning crash protection for airplane occupants and flight recorder systems were addressed to the FAA.

    These people are not screwing around.

    One thing to keep in mind while reading these incident reports is that, for the purposes of fact finding, some investigators will assign a single root cause (which, given the redundancy of modern aviation, is typically assigned to “pilot error”), while others cite contributory causes and situations. An incident investigator named Sidney Dekker wrote a gamechanging book about this called The Field Guide to Understanding Human Error, where he discusses that simplistically assigning blame isn’t just wrong, it’s actually counter productive, because the problems don’t get fixed. I picked up the paperback ($20 used), and I loved it.

  3. Read Medical Case Reports

  4. Speaking of macabre…I think it’s important to read medical case reports. Like I said, it sounds like it’s from left field, but it’s actually very relevant.

    The best site that I’ve found for this is The Journal of Medical Case Reports. It’s a publication dedicated to just that: case reports. They’re available for free, and they’re all written by practicing doctors who have documented interesting or unusual cases that they’ve encountered that they feel would be useful to their fellow medical professionals.

    Here’s an example: Low back pain during pregnancy caused by a
    sacral stress fracture: a case report
    (pdf). The medical jargon isn’t as important to us as the form and the function. Check out this abstract:

    Introduction: Sacral stress fractures are a rare but well known cause of low back pain. This type of fracture has also been observed as a postpartum complication. To date, no cases of intrapartum sacral stress fractures have been described in the literature.
    Case presentation: We report the case of a 26-year-old Caucasian European primigravid patient (30 weeks and two days of gestation) who presented to our outpatient clinic with severe low back pain that had started after a downhill walk 14 days previously. She had no history of trauma. A magnetic resonance imaging scan revealed a non-displaced stress fracture of the right lateral mass of her sacrum. Following her decision to opt for nonoperative treatment, our patient received an epidural catheter for pain control. The remaining course of her pregnancy was uneventful and our patient gave birth to a healthy child by normal vaginal delivery.
    Conclusions: We conclude that a sacral stress fracture must be considered as a possible cause of low back pain during pregnancy.

    Read through the rest of the paper. The frank and open discussion about the case, the evaluation, all of it lends itself as an evaluation by a knowledgeable professional giving his opinions to his peers.

    The doctors who authored this piece are doing exactly what we should be doing. How many times have you encountered some strange condition or a case that seemed bizarre, but when you discovered the underlying cause, was preventable through easy checking early in the process, if only you had thought to do that?

  5. Write Reports of Your Own
  6. You had to see this coming.

    Now that you have two very excellent examples of reports generated by professionals in their respective fields, you should come to see that you need to start writing reports on the things you experience, too.

    Part of being a professional is taking part in the professional community and sharing your knowledge with your fellow practitioners. The strange things that you notice can end up being very helpful to other system administrators. By publishing your findings, you are legitimizing your experiences and your professional status, plus documenting your experiences for posterity, plus encouraging others to do the same. There is absolutely no downside.

    So where to publish? Every conference that I know of is actively seeking practice and experience reports, which would cover either of the above types of reports. If you had a failure that you ran a post-mortem on, or you encountered an issue during the otherwise routine operation of your infrastructure that others could learn from, then by publishing a practice and experience report, you help everyone.

    There is also a journal for system administrators called ;login:, and they are always looking for case studies, which is what this kind of report would be.

Doing these things will not turn you into an amazing system administrator overnight, but taken together, they’ll provide excellent models of professional behavior that we should all work to emulate.

You don’t need to do any of these things to be good at your job, and I don’t want to give that impression. What I would like you to consider is that, given two equally proficient administrators, I will always prefer the one who has tried to share what he or she has learned rather than kept it to themselves.

Today, on a very special Standalone Sysadmin…

It’s been over a year since I last used that headline, but I feel like its warranted in this case.

After taking a year off from the full-time system administration gig, I’m happy to announce that I’m ready to jump back in.

I have just accepted an offer to join David Blank-Edelman‘s band of Merry Men in the College of Computer and Information Systems at Northeastern University in Boston, MA.

My official title will be Networking, Virtualization, and Research Cluster Administrator. There’s still a copy of the job description on if you’re curious.

I’m set to start at the beginning of August. Until then, I’ll be in Ohio packing up things and tying together loose ends. The Columbus LOPSA Chapter is going to keep on going, and Warner Moore will be doing the organizing from here on out. He’s already been really active in the chapter, and I have no doubt that it will keep improvig.

So yeah, I’m back into system administration. While that alone is fun, I can’t describe how excited I am to work in this environment. Having known David for years, I know that working with him, Christopher Allison, and everyone else at Northeastern is going to be an amazing experience. It’ll be the first time in my life I’ve ever worked with a team of system administrators. I really enjoyed working with my junior administrator Ryan Salomon, but this will be an altogether new kind of experience, and it’s one that I’m really looking forward to.

So yes, the blog title is technically incorrect (again), but I’m still not changing it. Just think of it like I’m representing the standalone sysadmins out there in this new role with *gasp* peers.

I promise that I’ll continue to share the new ideas and experiences that I have in this job, and as always, if you have any questions or concerns, drop me a line.

Thanks for reading my blog for all of these years. I hope that I can continue to write stuff that you’re interested in reading and sharing.