A considered reply to Mark Burgess

Yesterday, my friend Will Dennis posted a thought-provoking thread to the LOPSA-Discuss mailing list regarding Mark Burgess’s opinion column from ;login:, “Is DevOps the Future of SysAdmin?“.

Here’s Will’s email:

Just getting caught up perusing some recent ;login: issues, and ran
across Mark Burgess’ article about DevOps[1] in the April 2013 issue. I
don’t remember a discussion of this on this list, but maybe I missed it
or something… Anyways, seems like a good topic of discussion. (FWIW, I
tend to agree with his thoughts, although I’m not sure LOPSA [or SAGE]
was trying to “unionize” anyone…)

“Organizations such as SAGE and LOPSA seemed to lose their way, too; by
trying to “unionize” the profession, they effectively sent the message
that sysadmins just felt poorly treated and underrepresented when they
could have led the march to modernize practices and be the heroes of IT
emancipation. In fact, the profession as a whole simply failed to adapt
to the needs of the rapidly expanding IT industry. Perhaps, if sysadmins
had taken on the mantle of responsibility for integrating into business
processes, that might have led to their rising up the pay-scale
automatically. But system administration has remained, for many, an
introverted gaming occupation. Now it needs to become a more disciplined
engineering profession. And history is in danger of repeating itself
with a new generation of junior admins and impatient developers working
with the cloud, or with new scripting frameworks for automation.”

Should LOPSA “lead the march” to DevOps? If so, how?

[1] “Is DevOps the Future of Sysadmin? Bemoaning the Failures of the
Sysadmin Profession
” ;login: Vol. 38 No. 2 (April 2013), pp 6-7

I’ve got a lot of thoughts about this kind of thing. My first post on the subject back in 2010 still pretty much nails how I feel about the movement.

I draw a distinction between what I see DevOps as (DevOps is an increased interaction and interdependency between developers and operations staff) and the things DevOps do (rapid release, continuous integration when viable, infrastructure as code, configuration management).

I think that DevOps is a great benefit for the shops that can use that interdepartmental relationship, but not every company can (because not every company has developers, and some shops don’t have ops). On the other hand, the technologies and methodologies that have arisen from the DevOps movement are unquestionably more sophisticated than the methods that most non-DevOps shops use, and I think that there’s nearly universal benefit from adopting those methodologies and technologies.

But back to Will’s post, he asked, ‘Should LOPSA “lead the march” to DevOps? If so, how?’. I think I answered his question, but only indirectly. Here’s what I said:

Hi Will!

Great topic! Here are my (extended) thoughts, reading through (and the standard disclaimer of “These are my views, not views of the Board or Org or anyone else” apply, of course). Apologies in advance for the wall of text.

> by trying to “unionize” the profession, they effectively sent the message that sysadmins just felt poorly treated and underrepresented

Every once in a while, someone will bring up the topic of unionizing IT workers, but it’s dropped each time because of the inherent trade-offs. I don’t recall any time in recent memory when unionization was seriously considered, or even considered to be promoted as a viable and desirable goal for the profession. Maybe something happened before my time (and I’m fairly recent compared to some people, having joined in 2009), but I don’t think characterizing us as trying to unionize the profession is really fair.

> the profession as a whole simply failed to adapt to the needs of the rapidly expanding IT industry

Agreed completely. As an industry, a large portion of us are still struggling to hit a mark that was set (and subsequently abandoned by those on the leading edge) years ago.

On one hand, there’s always going to be a long tail of technological sophistication trailing behind the “state of the art”, and that there is a significant portion of the industry still doing things in a less sophisticated manner isn’t indicative of failure.

On the other hand, there are, without a doubt, companies with technologically backward IT departments who are doing things “because they’ve always been done that way”, and when the IT department ceases to be an enabling force and begins to retard corporate agility, then we (or at least, that particular IT department) have failed.

> But system administration has remained, for many, an introverted gaming occupation

I’m honestly not sure what Mark meant by a “gaming occupation”, but I think that it can be agreed that there’s a large contingent of our fellow admins who are introverted. Those of us who attend conferences might be tempted to remark about how extroverted their fellow admins are, but remember that the people who attend conferences are the vast minority of those in the profession, and probably not indicative of the general “state of the profession”, so far as its constituency is concerned. There are around 850 members of LOPSA. There are 45,000 users of /r/sysadmin on Reddit. There are over 100,000 users on Server Fault, and there are over a million users in the Spiceworks community. The thousand-or-so system administrators who attend LISA are the tip of the iceberg. How many are introverted or not is, I would say, largely unknown.

> Now it needs to become a more disciplined engineering profession.

Yes, for the love of all things holy, yes. But there’s a problem. Actually, there are lots of (surmountable) problems, but here, as I see it, are the big ones:

1) Lack of leadership
These kinds of profession-wide changes don’t happen on their own, for the reasons I’ll be listing below. There needs to be a standard bearer around which people can rally (or, if not rally, at least follow). LOPSA is unique in IT Administration (in my knowledge) in that it purports represent and speak for the profession. Its membership certainly comprises the most visible members of the profession, and if there is any value in appeal by authority, LOPSA is ahead of any any other organizations that I know of.

I think the primary difference between LOPSA now and a successful LOPSA in the future is action. It’s not enough to say that you’re the leader; you have to actually lead. You have to take the dangerous road and risk things. You have to not let fear of failure or fear of falling short stop you from doing things that matter.

Once LOPSA starts doing things, it will, necessarily, divide the membership into people who agree with the direction and people who feel LOPSA is doing the wrong thing. Unanimity in a group as diverse as ours isn’t possible. There’s nothing that will please everyone, so the best anyone can do is what they feel is the right thing, and let things fall into place as they will.

2) Lack of academic rigor
Although I draw a lot of parallels between the progression of the profession of medicine and IT administration, academically speaking, IT administration is much more like engineering or architecture than medicine. There are medical research facilities that exist solely for the purpose of expanding the corpus of medical knowledge, and the field being researched provides its own professional application. That is, largely speaking, medical research extends the sophistication capable of its practitioners.

Contrast that to engineering schools, which largely don’t conduct research *on engineering*. Instead, engineering makes use of advances in materials science, geophysical research, aerospace research, and so on. Engineering is the application of disparate sciences, but done in a controlled and (relatively) standard, professionally-approved manner. It’s easy to see that a mature IT administration profession will follow this model, rather than that of medicine.

That being established, the primary difference between the engineering professions and ourselves is that there are standards in engineering, borne out of hard learned lessons over decades and centuries. Professional accreditation of degree granting institutions is well-established in engineering. It is nonexistent in IT administration.

Part of this is that we started a good 100+ years after the modern profession of engineering did. We’re a new profession, and we’re still finding our feet in a lot of ways. There is no widely established standard of education – indeed, there isn’t even a widely established nonstandard education. It would be a stretch to say that there are much more than a dozen academic institutions which grant degrees in anything that can be construed to be IT administration.

Personally, I think that it behooves LOPSA to work, not necessarily as a source of truth for the content, but as a bridge between institutions who seek to develop and improve programs in this arena. Eventually, anyway. In the immediate short term, I’m not certain that we, as an organization, carry the gravitas to effect the change we want to see. But eventually we will, and personally, I think we should.

3) Lack of codification
Several professions have established a Body of Knowledge(BoK). Interestingly enough, there IS a System Administration Body of Knowledge. Geoff Halprin (a member of USENIX and founder of LOPSA) compiled it circa 1997. Several of the people reading this email already know this fact, and I would estimate that they are probably 99% of the system administrators in the world who are aware of it. Personally, I don’t know if it was complete at the time of its compilation. I know that it has not been maintained by the profession, or added to by any groups, and I’m unsure as to exactly which group should be the party to maintain it.

Nevertheless, our profession does not have a current BoK. I have heard the argument that it moves too quickly to maintain a BoK, but I’m unconvinced that it moves faster than Software Engineering, and that profession maintains the SWEBOK (albeit, the current SWEBOK is from 2004, v3 is under current development).

4) Lack of cohesion
This is maybe the most annoying problem from my perspective. It’s similar to the nagging, unanswered questions left by the Drake Equation. If you’re unfamiliar, the Drake Equation is a method of estimating the number of communicative extraterrestrial civilizations in the galaxy. It relies on a large number of unknowns, but the end result is that all but the most pessimistic interpretations end with the equation pointing to thousands of civilizations waiting to be found. The premier question is always, “Where is everybody?”

If there are 100,000 IT administrators on the internet active enough to ask and answer a question on Server Fault, where are they? Do they know that LOPSA exists and don’t care or aren’t interested or don’t feel like we offer anything? Or are they completely unaware of us? How do we find out the answer to that question? How do you reach these thousands and thousands of people that we are trying to represent?

There is absolutely *no* cohesion to our profession. There are, in essence, maybe a couple thousand administrators who are part of a profession, and tens or hundreds of thousands of people doing a job, that just so happens to be the same job the profession does. How do you convert those people? How do you make them aware, and offer them something of value, and convince them to join you?

That’s the big, unanswered question, and until we have an answer for that, we’ll never reach our full potential.

So there, my giant wall of text. Apologies for the length, but it’s something that I think about quite a lot.

So there you go. Feel free to comment on the topic. I’m definitely interested in your viewpoint.

  • Nice reply. Of all the aspects you addressed, the part that stood out the most for me was the lack of cohesion. I wonder if one factor may be related to the personalities the profession tends to draw.

    There’s the stereotype of the systems administrator sitting in the basement, shunning interaction with “normal” people and being happy in their own introverted world. In my interactions I’ve found that while the stereotype tends to be overblown, there isn’t really a sense of camaraderie or priority on helping teach each other to improve. Instead I get the sense that there’s a competition to prove how much more one person knows over another, and if you’re not good enough, then you shouldn’t be in the profession.

    I’ve thought in the past that these personality traits are where the idea comes from that your average “IT person” is unapproachable by others in the business. I’ve heard IT people lamenting, “We can’t fix what we don’t know is broken! What do they mean it hasn’t worked in a month?!”

    …well, they don’t want to talk to us. They find it a hassle, I’ve had some people say they feel stupid when talking to the department,…it’s hard to find people in IT that don’t carry this aura of superiority over non-IT (or junior IT) people.

    My experiences are only my own and of course aren’t a huge sample to draw conclusions from. But I think that if, generally, these observations are accurate, it may factor in to why you don’t see more cohesion and cooperation among sysadmins. I’d think if they can’t work on making themselves accessible to non-IT people (more than just knowing where your office is or how to submit a trouble ticket) then it makes sense to think this would spill over into their interactions with each other. Possibly with a little more hostility if they feel their knowledge or skills are being questioned.

    What do you think?

  • Dominic L

    Your point about lack of cohesion is an important one – in part it is a symptom of how “young” system administration is compared to other fields. We don’t need to wait centuries or millennia (how old is the field of medicine?) to adapt the benefits of organization that plumbers, electricians, and other trades have developed over the ages.

    One way to expand is for LOPSA members to each take up advocacy of the professional organization. We’re all posting on server fault, linuxquestions, technet, on our blogs, and so on. We can each end every post with something to promote the organization. Akin to the Linux Counter tags people seem to wear with pride (maybe that’s not cool any more, sorry for being so 2005) or otherwise.

    This shows leadership by LOPSA members in contributing to the organization, it shows leadership in growing LOPSA as a predominant organization. Cohesion and academic rigour, isn’t expanding the organization key in collecting the knowledge and getting an expanding the sample of sys admins necessary for this?

    Join the growing League of Professional System Administrators today! Open to anyone who wants to share experience with their peers!