Free Advice Is Worth What You Pay For It

It’s weird how things come in groups, seemingly. I can go for weeks or months and not get an email or message from readers asking advice, then a few will pop-up all at once.

In the past week, I’ve gotten no less than three messages, all from different people who are going through certain decision-making opportunities, and asking how it was that I got into system administration, and wanting to know how I gained the skills that I did.

If there’s anyone else reading who wonders about this, or is looking for an example of how someone learns system administration, then maybe this will suffice.

Note: this was a response to someone about to graduate in three weeks with their BS in Information Systems. They currently work for a webhosting company and handle a decent amount of Linux. They’ve been in both Windows and Linux shops, “done a bit of everything, run ESXi servers at my house with 2008 R2 AD, WSUS, FreeNAS, etc”. They’re getting ready to choose between two jobs, an easier one and one they’re afraid might be more than they can handle.

I learned almost entirely from books and from working on the products, just like you.

I went to college for a year (well, sort of – I went to DeVry before I figured out that it was a scam) but it taught me nothing relative to what I do right now. I learned Linux on my own by running it on my desktops over the course of a very long time (16 years so far).

My first IT job was in my first year of college doing remotely outsourced technical support for the dialup ISP. Then I moved to a different project, doing dialup technical support for MCI Worldcom (again, outsourced to the same company that BellSouth used).

When I moved back to West Virginia where I grew up, I got a job at a local ISP doing tech support, and I just happened to know Linux better than either of the two administrators who were implementing it in the server infrastructure.

They slowly “adopted” me into being a junior administrator-type. They taught me a little bit NT4 domains, and by the time I was an “official” junior admin, there was only one senior admin, and he and I taught ourselves Windows 2000 while I taught him more Linux. I eventually took over after he got a better job.

Back in those days, I tried to learn, but I took a kind of lackadaisical view of everything. I didn’t push like I do now. I just kind of went with things, implementing stuff as I needed, and I wasn’t anywhere near a “senior” sysadmin, I was probably a SAGE Level 2 when I left to move back to Columbus to take a programming job.

It didn’t take long before I found another sysadmin position, and my “stats” at that point were high-skill-level Linux admin, low-to-medium-skill-level Windows admin, and I was not incredibly well versed in the good overall tao of system administration.

You can look at my path of becoming a better system administrator in two ways: my skillset and my attitude. The second is, in my opinion, more important than the first.

What really turned the corner for my outlook was that I had started a thread on a forum that I frequented devoted to the “interesting command-line of the day”. Every day, I would post an interesting hack that I would develop using bash. I did things like write a command-line RSS reader in a bash shell, or report the most-recent earthquakes in the world over a 5.0. Just weird little things that I thought were interesting.

I took to writing longer entries (or diatribes about technology) on my LiveJournal, but one day, a friend of mine basically said, “look, it’s great that you’re writing this here, but no one knows what you’re talking about”. So I thought, “maybe I’ll make a blog”. So I made a blog to share all of my technical writing.

This introduced me to the sysadmin blogger community, where I linked to (and read all of the posts by) people like Bob Plankers, Jeff Hengesbach, Michael Janke, Saint Aardvark the Carpeted, and Ben Cotton. And I really do mean that I read everything they wrote.

I became aware of people like Tom Limoncelli, Ben Rockwood, and John Allspaw.

I kept writing, and got linked to by a lot of people. A weird group called LOPSA-NJ added my blog to their RSS page. I checked it out, and in February of 2009, I became a member. It seemed natural to go to the meetings when I moved to NJ, but they were a long way away. New York City was closer, so I started my own chapter there. Then when I moved back to Columbus, I started a chapter here, too.

In terms of my skillset, when I started at my job in Ohio, I was fairly well versed in all of the technologies they were using. They had a more involved backup plan than I was used to, and more physical sites, and way more data to move around. I had to begin thinking about data *en masse*, as opposed to in small, discrete files.

My boss there, the CTO, functioned (and still does) as the defacto DBA, and I learned the operation and administration of Oracle and PostgreSQL from him, from scripts, and from breaking it and fixing it.

Across the board, I worked to implement the ideas that I had been reading online in my own infrastructure – automation, separation of duties, structured management, repeatability. I learned more about backups from reading people like Curtis Preston and Preston DeGuise, and I worked to learn more about the underlying technologies.

Rather than working really hard to implement things the way my boss wanted, I began asking myself why we were doing things certain ways. I began to think about what my pain points were, and what caused them, and I began considering other ways.

We were an all-Linux shop, and we ran Slackware everywhere except on the desktops, which were split evenly between Mac and Windows. One of the things I hated was user account administration. Changing a user’s account on all machines, plus the pertinent Samba servers, plus ftp, etc etc really sucked. It got to the point that Moves, Adds, and Changes (MACs) were taking up most of my time. Then I found Likewise Open, and it changed my life. It allowed a Linux machine, almost ANY linux machine, to become part of an Active Directory infrastructure. We were a Linux shop, but I got a demo copy of 2003 Server, built an AD infrastructure, got some CentOS test machines up and running, and authenticated them against the directory. Then I showed my boss.

One of the things that Likewise-Open couldn’t do was run on Slackware, mostly because of the lack of PAM and package management. That meant migrating our entire infrastructure over to something that could work, which in this case was CentOS, since I had a very small amount of experience with it.

That was one example of implementing a new technology with a small amount of previous experience. SANs were a different story.

We had DAS on our file server, DAS on our backup file server (manually rsync’d between them), DAS on our database servers, web servers, etc etc etc. We didn’t use virtualization at all.

Because of the growth of data, we were going to need a new storage solution, and because of the age of our servers, an infrastructure refresh was called for as well. We were primarily a Dell shop, so naturally, we reached out to Dell to see if there was a deal they could cut us.

As it turned out, the 1955 Blade Servers were being EOL’d, and Dell was willing to cut a ridiculous deal to unload some on us. Add into that the two Dell-branded EMC AX4-5 SAN storage arrays that we needed, and we got an entirely new infrastructure for quarters on the dollar. And as a bonus, I got to build them in parallel with our existing infrastructure, so I could take as much time as I needed, within reason.

I’d never dealt with blades before. Or SAN storage. So I read as much as I could about them in documents from the manufacturer. You know those CDs full of docs that come in the boxes, but everyone always throws away? I actually put them in my CD-Rom and read them. And what I could find online. I searched for other people running them and read those blogs. I started reading some of the dozens of storage blogs that are out there talking about SAN storage. I took in as much information as I could find, and since I had my own lab full of this stuff, I built and played and broke and fixed and learned as much as possible. Then I implemented.

Your story won’t look like mine – no two stories ever look alike. That’s why the attitude and outlook is so important. A good healthy approach to this job will allow you to reach your maximum potential, and will help toward you not hating the job.

Always take the long view, and think about the ramifications of the decisions you’re making. Don’t be afraid to over-reach from time to time, but try to balance that out by knowing what your limitations are, and knowing that you can always exceed them by learning and by trying.

Deciding what job you’re going to take is always difficult. I’ve never once regretted going with the harder or more interesting choice, whenever I’m presented with two. I tend to be aggressive, because I have faith in my ability to grow into whatever I find myself in. But, at the same time, I always represent myself as I am, not as I want to be…that helps people underestimate me.

I don’t know if you’re biting off more than you can chew, but if you don’t try, then you don’t know either.

  • Chris

    Thanks for the article. Always interesting to see how others got their start.

  • Tim Kitchen

    I really like this entry, probably my most favorite one to date.

  • One of the best article i’ve read the last month.
    Plz keep up with similar articles.
    May inspire others to write similar articles or even a series of them.

  • This entry needs an upvote button. Good article, Matt.

  • Pingback: Happy Blogiversary Standalone SysAdmin | Standalone Sysadmin()