Admin Heroics

Date June 19, 2008

You know, 99% of the time, we have a pretty boring job. Sometimes we get to work on interesting problems, or maybe a system goes down, but for the most part, it's pretty mundane.

Sometimes, though, we get called to do relative heroics. Before I was even an admin, I did tech support for an ISP in West Virginia. Once, the mail server went down hard. 20,000 people around the state suddenly had no email, and the two administrators weren't able to be contacted. I was the only guy in the office who knew linux, and it just so happened that I had the root password to that server because I helped the younger admin a few weeks earlier with something.

I reluctantly agree to take a look at the thing, having never touched QMail (ugh), I delved into the problem. Numerous searches later led me to conclude that a patch would (probably?) fix the problem. I explained that to my bosses, and that I had never done anything like this before, but that I thought I might be able to do it without wrecking the server.

They gave me the go-ahead since we still couldn't contact either admin, and the call queue was flooded with people complaining. I printed out the instructions from the patch, downloaded it to the mail server, and applied it as close to the instructions as I could manage. Then, I started the software. It appeared to run, and testing showed that indeed, mail was back up.

I was a hero. At least until the next day when the main admin got back. Then my root access was taken away. Jerk.

Sometimes, we're called upon to extend beyond our zone of comfort. To do things that are beyond our skill levels, and to perform heroics under dire circumstances. These are things that make us better admins. Learning to deal with the kind of pressure that 20,000 people's programs aren't working and it's up to you, or that electricity is down and $18 billion dollars worth of financial reports aren't getting published and only you can fix it. Maybe it's that your biggest (or only) client had a catastrophe and you're the one handed the shovel. Whatever it is, it's alright to think of yourself as a hero.

Because that's what you are.

  • Bart

    Amen! I'm a developer myself and when our (custom) electronics was tested with my software, I felt a rush just standing there and watching the software chug away.

  • Matt

    It's a great feeling to watch something you created work like that. I'm glad I'm not the only one who gets that feeling!

  • Anonymous

    Aye, dude I miss the old days of a large 900mhz phone with an ear piece.. and the ultimate Hacky Sack!!

  • Kevin DeGraaf

    Are you saying that a qmail server suddenly and randomly died, and that a software patch was required to restore operation?

    Could you provide more detail? qmail is definitely strange, but I've never found it to be unreliable.

  • Matt

    Kevin,

    I'd love to provide you more details, but this was in 1999 or 2000. My recollection of the exact error isn't too good.

    I do know that I applied a corrective measure online, and I seem to recall it being a patch, but I wouldn't be able to swear in court about it. A couple of years later, we replaced it with CommuniGate because of some of the bad experiences we had with QMail.

  • brandonk

    That and the QMail admins were complete crap.

    =)

  • Steven

    Heroics for single incidents from time to time....good. Having your whole organization operate off hero syndrome with every project being a death march....bad. Don't rely on always being a hero. Be patient and hold responsible those that are responsible for the work. Note, I'm NOT saying you did the wrong thing.

  • Matt

    Steven,

    Absolutely right. From time to time, most systems need righted, and in a hurry. Having to do that too often leads to poor planning and cruft (not to mention massive downtime when it catches up with you).

  • Ruben

    In university, we have a class were a lot of seperate teams design different components for an autonomous train system. Some people create a wireless network card for communicating with the train, some create voice recognition software, some create the power supply, etc...

    It's a great rush to see that little train do what it's supposed to do, the work of 20+ people coming together and doing what you want it to do. That's the rush I want in my future job: seeing something work because you put a lot of effort and teamwork in it.

  • robert

    being an 'accidental' hero is good.. however, @work i've had a colleague which seems addicted on being a hero... being a developer on hi-priority project (by hi-prio means some top hats require feature a, b and c to be available within three days.. leaving analyst awake for 2 whole days).. regardless the spec, this addict made up his own logic.. analyzing on his own (means delaying progress).. shows up very late whenever bugs found..

    i don't know of whether he is addicted on being a hero or being addicted to crisis situation..

    he was a great programmer... but in the end he was let go by the management...

    being a accidental hero is ok, but beyond that its either the management is incapable, or the hero is self is a sick puppy