And when you gaze long into an infrastructure, the infrastructure will gaze back into you.

You have to create your life. You have to carve it, like a sculpture
William Shatner

Fortunato, from The Cask of Amontillado, by Edgar Allen Poe

There is no form of interaction with the world that does not change it, in some way. Even the act of observing alters the observed. It should be no surprise, then, that when we set out to change or build anew, that we leave the world in a dramatically different fashion than we found it.

But what is changed in us? How are we affected by our actions? Hopefully we learn. Improve. With practice comes rewards, and the sweat of our labor yields results. But those results aren’t always positive, and sometimes it’s hard to know the good from the bad. I’ve often made the mistake of being proud of failures, when I should have been ashamed, had I known then what I found out later. I think this is part of the learning process.

I have wanted to write this blog entry for two years. It’s an idea that I’ve never put into words until now. I wanted to tell you then, but I didn’t know how. I think I might have learned enough to do it justice.

We build things, you and I. We, like sculptors or masons or carpenters or architects, build and design and create. We do it with as much wisdom as we have, making such decisions as we are able to based on who we are at the time, and the resulting infrastructure is a reflection of ourselves, as much as it is a system. We built it in our image, whether we meant to or not.

In small infrastructures such as I’m used to, I built lots of things. I wasn’t just responsible for the proverbial town church, it was the church and the stables, the meeting hall and the inn. I built it all, and I left my signature wide and bright for anyone who could read it. I built things the way that I thought they should be as often as I could, and while they weren’t perfect, they were mine, and I was prideful of them, in my own way.

When I left, the search for my replacement took a long time. Much longer than I’d have preferred, and to an eventually unsatisfactory conclusion for everyone involved, I believe. I contented myself with the knowledge that my skill set was sufficiently wide in breadth and complex in nature that I was hard to replace. I used this to buoy my ego. Although I had sympathy for the people I was leaving, and the one I left in my stead, it felt good to be needed and wanted, and I was proud that I could fill that role like no one else we’d found.

Such is the folly of the unwise, I’m afraid.

My friend Shawn asked me today how things were different being in an educational institution versus being in a small business. We talked about different aspects of the environment, many of which I’ve discussed here on my blog.

One of the things that came up was the concept of silos, or a division of labor where each person has a few distinct areas that they concentrate on.This kind of specialization is common in organizations with more resources than small businesses, and it has several profound consequences, many of which aren’t obvious at first glance.

The first of which is the deep domain knowledge that one can obtain (and in many cases, needs to obtain). The analogy that I use most frequently is as follows: Imagine yourself sitting on the floor in an infinitely large room, filled with infinitely many buckets. Each hour, you are given a golf ball to place into a bucket. What do you do?

Some people would start immediately filling the bucket closest to them. Every hour, they’d place their golf ball in that bucket, and over time, it would get more and more full of golf balls.

I, as I suspect many of you, take a different approach. I spend my golf balls in as many buckets as I can. Some buckets get more than others, but there aren’t many empty buckets within the reach of my throw. Over the years, a lot of the buckets have gotten relatively deep. Those are my favorite buckets, and even though they’re deep, they don’t come close to someone who picked that particular bucket to constantly deposit their golf balls. Remember that the next time you feel good about knowing more than someone. They just spent their golf balls differently than you.

So by spending our time doing a few specific things, we gain a much more thorough degree of knowledge in those specific arenas. We can afford to become experts because we’re going to be doing it anyway. We practice and learn and work and get better by throwing golf balls in that bucket, and the bucket gets more and more full.

By specializing, we get better at what we do, even though we do fewer things. Because we do fewer things.

With small infrastructures, we build the world around us according to our knowledge and experience – which buckets are more filled. In a large, specialized environment, we are changed to meet the needs of the environment. We fill the buckets that are related to our role, not the other way around.

Another outcome of this type of situation is that over time, two people filling the same buckets tend to arrive at the same place. That is not to say that there is one specific way to do things, but that the waypoints are the same and that there is a commonality in tools and knowledge. This leads to a situation where any person who has put a sufficient number of golf balls into that specific bucket can perform the tasks required by that position. The administrator largely has not altered the roles and responsibilities of that job, but the position has altered the admin (and as a consequence, dictated the form of those people who can take that position in the future).

Where I made the mistake of forming my infrastructures around me, specialized to my particular desires and skills, I found myself impossible to replace. If I would have known better from the start, I could have crafted the infrastructure using a more common pattern. Hiring enough people to specialize was outside the realm of possibility for me, but there are certainly things I could have done to make it easier to find a replacement. I bricked myself in. I was Montresor and Fortunato at the same time.

The advice that I’ve wanted to give for two years, but didn’t know how to say it until now, is that the infrastructure should be the mold, not the sculpture.

Thanks for reading. I hope you enjoyed it.

  • John M

    Thanks for the Post, Matt.

    We all forget that what we create is part of a greater organization, and want to create it in our own limited scope. It is good to remember to that we are only as good as the documentation we leave behind.

  • Fred Woodbridge

    If this is the result of two years’ cogitation, I shudder to think what it’d have been in five! :p

    By which I mean, I’m trying to wrap my mind around it and failing. If, to use your metaphor, the room is infinitely large and the number of buckets is also, the question wouldn’t be which bucket to fill, but why? Perhaps the infrastructure molds you, rather than you it?

  • Danielle

    Very well said. In the past, working in university environments, I had the breadth but often struggled for lack of the depth; quite often a project really needed me to have more in-depth skills on the topic than I could budget the time to gain. Today I work in a corporate environment where I have far more depth. It works for the service we are providing (software as a service of my employer’s business intelligence products.)

  • Fred: Both you and the infrastructure are molded by each other, but certain environments place constraints on the types of people that can fill the roles, thus molding those who do fill the roles.

    When the infrastructure is the sculpture, and you the artist, you build it to your designs and abilities, whatever they are at the time of creation, but if you follow a well-tested design and allow it to guide your way, you are the sculpture, created by the mold that the infrastructure creates for you.

  • Lord_NShYH

    Thank you.

    I believe the master sculptor, in creating their mold, must be acutely (and intimately!) aware of the negative space used as a means to provide measurement of the positive space of their creation.

    The sculptor may chisel away, and they may yet cast their molds.

  • clouseau

    I’m struggling to understand this post. Do you mean that in your past work you should have used VMWare instead of, e.g., ProxMox because it is more common and replacement admins are more easily found? That single-person IT operations should stick as close to the mainstream as possible?

  • Nathan Powell

    I agree. It’s similar to the “hit by a bus” maxim. In other words “if you were hit by a bus tomorrow could another competent admin walk up to this console and understand what you did”.

    However, I think in all organizations there will always be a certain amount of this sort of debt because we are all human. We make decisions, with the best of intentions, that will be confusing to others. However, you’re right, we should try to minimize that.

  • Wasn’t your previous infrastructure RHEL on VMWare backed by an EMC array? With some puppet brought in toward the end? That seems…pretty standard to me. I’m sure you had some hacked together scripts (who doesn’t?) but it also sounds like you’re being needlessly hard on yourself. I thought the point of “standalone sysadmin” was that such a role isn’t easy.

  • Travis

    Your conclusion needs revision. The standalone sysadmin (the person often choosing the infrastructure) shouldn’t be both mold and sculpture. The goals and expectations of the employer should be the mold, not the ‘infrastructure’.

    Isn’t the whole article a contradiction? While explaining your lesson learned about the importance of not “bricking yourself in’, you continue to “brick yourself in”. The article is very creative but unnecessarily sophisticated. Why not save us both some time and simply tell us you learned the importance of following commonly recognized best practices? Working and writing in ways that are beneficial and easily followed are keys to being replaceable.

  • Pingback: Leaving the LOPSA Board | Standalone Sysadmin()