Instrumentation.

Date November 12, 2013

Yesterday, I briefly discussed some of the events I took part in at LISA13 in Washington DC, but I wouldn't say it was an exhaustive list of everything I did. You see...I also played hookie on Monday, rented a car, and drove some people to the Udvar-Hazy Center, otherwise known as the Air and Space Museum Hanger.

Every time I've ever been to the Smithsonian Air and Space Museum, it's always the one on the National Mall. They have a lot of really cool stuff there, like Apollo landers, the Spirit of St. Louis, Spaceship One, and hundreds of other air-and-spacecraft. But the big birds are reserved for the Udvar-Hazy center, because there's no parking spot on the National Mall for a Concorde. Or one of these.

Yep, they have a Blackbird. Oh, and right behind it? This girl:

That's the Space Shuttle Discovery. No big deal, right? Seriously, if you're visiting DC, you must make it out to Udvar-Hazy. It's so worth the trip.

One of the other things they have is a selection of space capsules. I've been to enough museums now that I've seen plenty of them. Before, I'd concentrate on the tininess of the cockpit. How would you like to be shoved into that sardine can for days or weeks, flying through space with a limitless vacuum just outside the metal walls?

Maybe it's the fact that I've been playing a lot of Kerbal Space Program lately, or maybe it's the fact that I was experiencing things by visiting with a group of sysadmins, but for whatever reason, instead of concentrating on the cramped quarters, I opened my eyes and I realized that even the tiniest space capsule has a TON of switches, knobs, screens, and dials. The instrumentation is amazing!

Now, we see things like the Dragon from SpaceX, and yeah, it's complicated because it does a lot. It's modern and docks with the ISS, and all of that, but what about the first ones? Suppose you looked at the very first space capsule that John Glenn rode in. How many controls and dials would you see? How many things do you think a space capsule does?

You know it has to control the angle of re-entry, so you could expect some kind of control stick arrangement for that. It probably should monitor fuel, so you'll need a dial for that. Temperature, both internal and external would be useful. Also an altimeter, and an emergency "pop the chute" button in case the chute didn't do that on its own. Plus the wheel you'd unscrew to open the hatch. If I were imagining it in my head, that's probably what I'd say. I imagine I'm simplifying a little bit, but I don't think I'd be that far from reality.

Except I'm dramatically wrong. Here's John Glenn in the cockpit of the Friendship 7 capsule:

John Glenn in control of the Friendship 7

When I saw the capsules at the Smithsonian had instrumentation like that (and more!), I was kind of blown away. What do all of those things do? (As it turns out, there's an app for that) But it's just simply amazing. Every single function that could possibly be executed is instrumented there, with a physical switch or dial. That's kind of inspirational to me.

To some extent, modern IT has gotten away from these kind of command and control interfaces. Where, in previous times, it might have seemed advantageous to us to have a "command and control center", now, we largely try to automate away all of the interactive features and to get our systems to fly by wire, as it were. These switches and knobs have been replaced by API calls, and I'm much more comfortable with things like that.

But what of the dials? Feedback is still just as important as it was in John Glenn's day. Yes, we have alerting to those conditions that we know to watch for, and chances are good that you're trending certain things like bandwidth, load, and so on (partially because those are easy, and partially because they've been standard to trend since even before Tobi Oetiker wrote MRTG). But is that all you're gathering? Is that enough?

I keep reading through Etsy's amazing blog entry from a few years ago, Measure Anything, Measure Everything, and it kind of speaks to me. Diskspace is cheap. Information isn't. Why aren't I monitoring more? Why aren't I monitoring business metrics? Is it worth the effort of implementing? What's the potential payoff? What's the downside?

What I've eventually come to is that, for the first time in my professional life, I am creating an annual goal for myself. This year, I'm going to concentrate on instrumentation. I'm going to implement better, more thorough monitoring, I'm going to figure out what matters to me and my employer, and figure out what the best way to present it is. I don't know what's important and what isn't until I see it, so I'm going to be liberal in my data gathering.

One of the first things I'm going to be doing is standing up a Graphite instance that I don't hate. I spent a lot of the train trip back from Washington playing with it using Vagrant, and I wanted to help other people. Graphite has a reputation for being difficult to install. The amazing Jason Dixon wrote Synthesize to do the hard work for you, but it's limited to Ubuntu 13.04, and we're settled on 12.04 until the next LTS is released, so I cribbed some of his ideas and those of other HowTos and created a Vagrantfile that installs Graphite. It's not a vagrant box - it's a generic Precise64 box, but iif you look at the Vagrantfile, it walks you through the steps of installation, it just does it automatically and without any interaction.

You can take the automated steps and change them to meet your environment however you want. Hopefully it will help you set up Graphite in your own infrastructure. I'm still working on Collectd, and I may have some places to use something like statsd, too. But I'm making progress. If your monitoring environment needs help, maybe you can start now, too. It's just one step at a time, right?