After some discussion, my friend Brandon Burton and I started a podcast called SysAdministrivia. We released 2 episodes of the 4 that we recorded, but life intervened. Brandon changed jobs, I quit mine, and I moved to Ohio. And 18 months passed.

Now, though, we’re back. Episode 003 was released today. It’s only 25 minutes, so it should be easy to listen to. There are show notes with links on the website.

Please check it out, and if you like the format and idea, subscribe to the RSS. We’ll work on getting iTunes integration, but if there’s anything we should be doing that we aren’t, drop me a line or a comment below and tell me!

Thanks for listening!

I’ve had this blog post sitting in my queue for a week. I really thought this whole thing would blow over, but people keep talking about it, and as an interested conference-goer, it’s not outside of the realm of my interests…it’s just that I think that this whole “drinking at conferences” thing is pretty overblown.

Work is the curse of the drinking class
–Oscar Wilde

A lot of people go to conferences and drink while they’re there. A subset of people go to conferences SO that they can drink. Which, speaking economically, makes no sense to me. But whatever, there you have it.

Now, the post that I linked to above, Ryan Funduk’s Our Culture of Exclusion begins with two paragraphs about a culture of exclusion (and hey, I think that there really is a small group of individuals who actively try to exclude other people. I wrote Xenophobia and Elitism in the Community two years ago), but then he dives headlong into an accusatory diatribe over the “Alcohol Clique”.

Let me confront this label head on.

There isn’t an alcohol “clique” at tech conferences. There is a culture of technical people, and one aspect of that culture is alcohol. It isn’t the reason that (most) people come together, but it is something that is shared by many members of the community. That isn’t a clique, that’s mainstream.

Enough of the label, let’s actually get to the substance of the problem.

Ryan views drinking as exclusionary because some people don’t drink, don’t want to frequent the places where alcohol is drunk, and don’t want to be around people that have been drinking.

I sympathize with people who don’t drink, don’t want to frequent the places where alcohol is drunk, and don’t want to hang out with people who have been drinking. It has to be frustrating when 95% (99%? I don’t know the numbers) of a conference disappears in order to go partake in something you cannot or choose not to partake in.

At the same time, though, the activities of this vast majority are not exclusionary. You are not being excluded, you are excluding yourself if you choose not to be around people who are drinking. This isn’t a case of a disability barring you from entering a building – it’s a choice that you’re making not to be around people.

That being said…I agree with a lot of the observations that Ryan makes. There certainly are people (too many of them) that drink until they’re witless. Who are loud and obnoxious…and who frequent places that are loud and obnoxious. I don’t like that particular “scene”, so I don’t go there or hang out with those people.

Some of the things that go on around other conferences (like VMworld) are kind of…off the chart, as far as bad decision-making goes, so I’m going to exclude them and deal with the ones that I have more experience with, such as PICC and LISA…and it’s not all sunshine and roses, but it’s not all bad, either.

As you probably know, LISA has Birds of a Feather sessions, or BoFs, after normal conference hours. These are “unconference” sessions where people can create a meeting about a particular topic, and then (the theory goes) anyone with interest in that topic shows up and you can have an engaging and informative discussion. The BoFs are scheduled each night, Monday-Thursday until 10pm or so.

That’s exactly what happens, too. There are a lot of interesting BoFs on good topics that are all well attended. Until Wednesday, that is. Wednesday is when the actual LISA conference begins, with the previous two days having been training only. On Wednesday, the floodgate of attendees opens and the BoFs change.

Instead of having 6 rooms where each has 4 hours of BoFs full of people, you have 3-4 rooms of poorly-attended BoFs because enterprises grab up big chunk of BoF time…which I’m not complaining about. Vendors have the same right to BoF time as individuals (note that I don’t know if a BoF time block is negotiated as part of a sponsorship package or not, but it may be, since a couple of the vendors get 2 hour BoF-space). What I do complain about is that these things are called BoFs in the first place.

Look at the LISA’11 BoF schedule. Take note of everything that says “vendor BoF”. My impression, from the majority that I’ve been to, is that they are far-less BoF and far more marketing and headhunting opportunities, in some cases with a party atmosphere thrown in. There’s often very little actual discussion of the company’s technologies other than maybe a slideshow or a technical “state of the union”-type address.

If there’s so very little signal in all of the noise, why do people go? In a word…beer. At the very least, there are buckets full of beer at the front door of the room when you walk in. The higher-end ones have wine, too, and although I’ve never seen liquor in an official BoF, there very well could be. I don’t know.

The thing with the vendor BoFs is that the vendors see buying beer as either a community service of sorts, or a marketing expense, or maybe as a human-resources endeavor, but whatever the reason, it virtually guarantees that any BoF that runs up against, say Google’s, is going to have next to no one attending it.

But what are you going to do? As someone who throws a BoF or two every year, I have a choice to make. I can schedule my BoF before the big vendor BoFs start, or I can schedule it late enough that all of the vendor BoFs are over, or I can schedule it against what I judge to be the weakest vendor BoF in my time frame. I’ve done all of those. But let me be very clear – I am not being excluded. I am making a choice to not attend a vendor BoF and partake of whatever it is that they’re offering, be it alcohol, company, or ambiance. It is my choice.

Yes, the situation sucks from the perspective of someone who feels like they have something to offer people who are busy drinking somewhere else, but that is the situation at hand. Would eliminating vendor BoFs solve the problem? Hell no. LISA is held in the heart of metropolitan areas for a reason – good access to local restaurants, pubs, and other attractions. Enterprises would LOVE it if the LISA staff got rid of vendor BoFs, because hotels charge an astounding amount of money for providing beer, and it would be cheaper to rent an entire pub than it is to supply hotel beer and wine to the probably 400+ people who showed up to Google’s BoF last year.

If you eliminate vendor BoFs then you not only continue to keep vendors involved, but you actively draw people away from the conference location. This isn’t even “not winning”, it’s losing worse.

There is no way to win the war on an “anti-alcohol” ticket. Ask the 21st amendment of the United States Constitution.

So instead, winning the war means picking your battles and deciding what you want to fight against.

Here is what I am against, that Ryan referenced in his post. I am against parties that are so dark you can’t see the person you’re trying to talk to. I am against parties that are so loud that you can’t hear the person you’re trying to talk to. And I’m against parties that are so crowded that you can’t find the person you’re trying to talk to. If you’ve been to conferences, you know the parties that I’m talking about, and at LISA, I can think of at least one that was a “vendor BoF”.

So, if I were to take the stance that I was against a party of that sort, my first step would be to identify why people wanted to be at that party.

Is it the ambiance? For a few people, sure, but there’s far more ranting against that than praise of it. Since the people who like that “club” feel are going to seek it out, and my goal isn’t to be in a place like that, our differences on the subject are irreconcilable, so I’ve got to move on.

Is it the giveaways? Sure, that has a lot to do with it. If someone is giving away an iPad or an Android, or a pass to LISA next year, but you’ve got to be present to win, you’re going to have a much higher attendance than you would otherwise.

But I think that the big reason is the alcohol, because free beer is a big draw for people who like to drink, and large vendor BoFs have it in spades. It’s the biggest draw that I hear about whenever I talk to someone who’s going to those BoFs.

So, in a thought-experiment, how could you fix the fact that people are going to vendor BoFs instead of content BoFs? You could eliminate the alcohol and giveaway aspect, but we already established how that turned out. Or you could level the playing field.

If all of the BoFs had the same access to alcohol, snacks, sodas, etc that the vendor BoFs did, then the vendor BoFs would be on equal footing with the guy who wants to hang out with people and talk about shell scripting, or configuration management, or anything else that no vendor would ever sponsor on their own. If there were no incentive to attend a useless BoF, then you wouldn’t. And vendors COULD move away from the conference and have unofficial BoFs at local pubs, but they wouldn’t, for the same reason they don’t now – competition from other more convenient (i.e. on the conference site) BoFs that offer the same or similar levels of desirable features (i.e. alcohol).

Anyway, enough about conference BoFs, and back to the culture of drinking stuff.

Again, yes, our culture does include drinking as one of its hallmarks, but I have never, ever, not even once, been (or seen anyone) put down for deciding not to drink. And I know a lot of people that can’t or won’t drink. In fact…

I didn’t write about this here, because, well, I was “busy”, but a few weeks ago, during a Tech Field Day event, I had to go to the hospital because I was experiencing a condition called atrial fibrillation, which is where the top part and bottom part of your heart disagrees about when and how often to pump. It’s very uncomfortable, and can be dangerous if left untreated, so I spent a few days in the hospital.

One of the outcomes of that was, because they feel that my having alcohol was a contributing factor, they advised me not to drink to excess. I decided that three days in the hospital was enough, and so I decided that the easiest way not to have a second beer is to not have the first. I haven’t had a drink of alcohol since the night I went in to the hospital.

That’s a big change for me, because I’m not a heavy or regular drinker, but I really enjoy good beer and good scotch, but I’m making a decision not to have them any more (although I reserve the right to taste new and interesting alcohols, I’m not going to be sitting down and having one).

By the arguments of Ryan and several other people who have weighed in on the topic, I could now consider myself excluded from the events where drinking takes place. Hogwash.

Were I not to be around people who are drinking, it would be my decision. As it is my decision to drink or not, is it not my decision to share company with people or not? If I don’t like the place they’re hanging out, I can hardly say that I’m being excluded for my own preferences.

No, I’m not going to take this exclusionary view. Instead, I am going to continue to hang out at the BoFs (both with and without alcohol), I’m going to continue to go to the hotel bar after hours. And I’m going to continue to not drink alcohol, and I have every confidence that no one is going to give me a second thought over it. And if the people I’m with drink too much to the point that I, in my sober state, can’t stand to be in their company, I’ll leave and find someone else, or go sleep. But I’m not being excluded.

So there, that is my very long, very complaint-filled view on this culture of exclusion thing. Let me know how wrong I am in the comments.

In the past week, I’ve gotten no less than three messages, all from different people who are going through certain decision-making opportunities, and asking how it was that I got into system administration, and wanting to know how I gained the skills that I did.

If there’s anyone else reading who wonders about this, or is looking for an example of how someone learns system administration, then maybe this will suffice.

Note: this was a response to someone about to graduate in three weeks with their BS in Information Systems. They currently work for a webhosting company and handle a decent amount of Linux. They’ve been in both Windows and Linux shops, “done a bit of everything, run ESXi servers at my house with 2008 R2 AD, WSUS, FreeNAS, etc”. They’re getting ready to choose between two jobs, an easier one and one they’re afraid might be more than they can handle.

I learned almost entirely from books and from working on the products, just like you.

I went to college for a year (well, sort of – I went to DeVry before I figured out that it was a scam) but it taught me nothing relative to what I do right now. I learned Linux on my own by running it on my desktops over the course of a very long time (16 years so far).

My first IT job was in my first year of college doing remotely outsourced technical support for the BellSouth.net dialup ISP. Then I moved to a different project, doing dialup technical support for MCI Worldcom (again, outsourced to the same company that BellSouth used).

When I moved back to West Virginia where I grew up, I got a job at a local ISP doing tech support, and I just happened to know Linux better than either of the two administrators who were implementing it in the server infrastructure.

They slowly “adopted” me into being a junior administrator-type. They taught me a little bit NT4 domains, and by the time I was an “official” junior admin, there was only one senior admin, and he and I taught ourselves Windows 2000 while I taught him more Linux. I eventually took over after he got a better job.

Back in those days, I tried to learn, but I took a kind of lackadaisical view of everything. I didn’t push like I do now. I just kind of went with things, implementing stuff as I needed, and I wasn’t anywhere near a “senior” sysadmin, I was probably a SAGE Level 2 when I left to move back to Columbus to take a programming job.

It didn’t take long before I found another sysadmin position, and my “stats” at that point were high-skill-level Linux admin, low-to-medium-skill-level Windows admin, and I was not incredibly well versed in the good overall tao of system administration.

You can look at my path of becoming a better system administrator in two ways: my skillset and my attitude. The second is, in my opinion, more important than the first.

What really turned the corner for my outlook was that I had started a thread on a forum that I frequented devoted to the “interesting command-line of the day”. Every day, I would post an interesting hack that I would develop using bash. I did things like write a command-line RSS reader in a bash shell, or report the most-recent earthquakes in the world over a 5.0. Just weird little things that I thought were interesting.

I took to writing longer entries (or diatribes about technology) on my LiveJournal, but one day, a friend of mine basically said, “look, it’s great that you’re writing this here, but no one knows what you’re talking about”. So I thought, “maybe I’ll make a blog”. So I made a blog to share all of my technical writing.

This introduced me to the sysadmin blogger community, where I linked to (and read all of the posts by) people like Bob Plankers, Jeff Hengesbach, Michael Janke, Saint Aardvark the Carpeted, and Ben Cotton. And I really do mean that I read everything they wrote.

I became aware of people like Tom Limoncelli, Ben Rockwood, and John Allspaw.

I kept writing, and got linked to by a lot of people. A weird group called LOPSA-NJ added my blog to their RSS page. I checked it out, and in February of 2009, I became a member. It seemed natural to go to the meetings when I moved to NJ, but they were a long way away. New York City was closer, so I started my own chapter there. Then when I moved back to Columbus, I started a chapter here, too.

In terms of my skillset, when I started at my job in Ohio, I was fairly well versed in all of the technologies they were using. They had a more involved backup plan than I was used to, and more physical sites, and way more data to move around. I had to begin thinking about data *en masse*, as opposed to in small, discrete files.

My boss there, the CTO, functioned (and still does) as the defacto DBA, and I learned the operation and administration of Oracle and PostgreSQL from him, from scripts, and from breaking it and fixing it.

Across the board, I worked to implement the ideas that I had been reading online in my own infrastructure – automation, separation of duties, structured management, repeatability. I learned more about backups from reading people like Curtis Preston and Preston DeGuise, and I worked to learn more about the underlying technologies.

Rather than working really hard to implement things the way my boss wanted, I began asking myself why we were doing things certain ways. I began to think about what my pain points were, and what caused them, and I began considering other ways.

We were an all-Linux shop, and we ran Slackware everywhere except on the desktops, which were split evenly between Mac and Windows. One of the things I hated was user account administration. Changing a user’s account on all machines, plus the pertinent Samba servers, plus ftp, etc etc really sucked. It got to the point that Moves, Adds, and Changes (MACs) were taking up most of my time. Then I found Likewise Open, and it changed my life. It allowed a Linux machine, almost ANY linux machine, to become part of an Active Directory infrastructure. We were a Linux shop, but I got a demo copy of 2003 Server, built an AD infrastructure, got some CentOS test machines up and running, and authenticated them against the directory. Then I showed my boss.

One of the things that Likewise-Open couldn’t do was run on Slackware, mostly because of the lack of PAM and package management. That meant migrating our entire infrastructure over to something that could work, which in this case was CentOS, since I had a very small amount of experience with it.

That was one example of implementing a new technology with a small amount of previous experience. SANs were a different story.

We had DAS on our file server, DAS on our backup file server (manually rsync’d between them), DAS on our database servers, web servers, etc etc etc. We didn’t use virtualization at all.

Because of the growth of data, we were going to need a new storage solution, and because of the age of our servers, an infrastructure refresh was called for as well. We were primarily a Dell shop, so naturally, we reached out to Dell to see if there was a deal they could cut us.

As it turned out, the 1955 Blade Servers were being EOL’d, and Dell was willing to cut a ridiculous deal to unload some on us. Add into that the two Dell-branded EMC AX4-5 SAN storage arrays that we needed, and we got an entirely new infrastructure for quarters on the dollar. And as a bonus, I got to build them in parallel with our existing infrastructure, so I could take as much time as I needed, within reason.

I’d never dealt with blades before. Or SAN storage. So I read as much as I could about them in documents from the manufacturer. You know those CDs full of docs that come in the boxes, but everyone always throws away? I actually put them in my CD-Rom and read them. And what I could find online. I searched for other people running them and read those blogs. I started reading some of the dozens of storage blogs that are out there talking about SAN storage. I took in as much information as I could find, and since I had my own lab full of this stuff, I built and played and broke and fixed and learned as much as possible. Then I implemented.

Your story won’t look like mine – no two stories ever look alike. That’s why the attitude and outlook is so important. A good healthy approach to this job will allow you to reach your maximum potential, and will help toward you not hating the job.

Always take the long view, and think about the ramifications of the decisions you’re making. Don’t be afraid to over-reach from time to time, but try to balance that out by knowing what your limitations are, and knowing that you can always exceed them by learning and by trying.

Deciding what job you’re going to take is always difficult. I’ve never once regretted going with the harder or more interesting choice, whenever I’m presented with two. I tend to be aggressive, because I have faith in my ability to grow into whatever I find myself in. But, at the same time, I always represent myself as I am, not as I want to be…that helps people underestimate me.

I don’t know if you’re biting off more than you can chew, but if you don’t try, then you don’t know either.

TELNET had some good things going for it — a local-echo mode and a well-defined network virtual terminal. Then SSH came along and added minor enhancements like confidentiality and authentication, at the cost of losing the local-echo mode and the well-defined terminal semantics.

Terminal emulators are a hard sell. I mean, yeah, we want that functionality, but at the cost of selling our souls? OK, if not our souls, how about our sanity?

Ever had a misbehave program freak out and corrupt your display? Or maybe you’d cat the wrong binary, then suddenly f?Kfjd!ܾ!j?sRc?CH?U^, it looks like your mom picked up the phone during your BBS session, and you have to type ‘reset’ to fix it. It’s not world-ending, but it is irritating.

As amusing as the telnet quote is, it’s right. Local echo is nothing if not immediate. We get feedback. Granted, in the case of telnet, that feedback doesn’t mean a whole lot, but still, it’s nice to be able to type and not see the letters show up several seconds later just because you’re typing into a terminal from 35,000ft using an internet connection that cost you the price of a month of dial-up back in the days when mom could interrupt your session of Legend of the Red Dragon.

Enter Mosh, a new take on terminal emulators.

Mosh is a client/server model. You run it almost exactly like you would ssh:

mosh msimmons@servername

It actually connects over ssh, then runs mosh-server on the other end. That’s when things get a little strange.

Mosh actually performs its transmissions via encrypted UDP datagrams. In the spirit of the heathen connectionless-UDP-protocols, it also doesn’t try to deliver each and every individual packet that the server spits out – so when you screw up and run find /, you don’t have to wait 10 minutes for your futile ctrl-c keystrokes to get to the server. Plus, and maybe most damning, it allows you to close your laptop, get off the plane, go into the terminal, pay ANOTHER $20, and connect back in immediately, all without running a screen session. Clearly, this is the work of Satan.

In all seriousness, though, this is pretty handy. What they actually do is, instead of treating the terminal like a stream (which is what every command line terminal emulator ever has done), they treat it like a state machine, and the network transmissions are all about bringing the client state into sync with the server state. That’s a cool idea.

It uses AES-128 in OCB mode. Of course, the Layer-4 headers are still unencrypted, but IP Over Unicorn doesn’t exist yet. *ahem*

Check it out. It’s free, available for a lot of systems, and the formal paper will be presented at this year’s USENIX ATC’12 conference during Federated Conferences Week. Cool, right?

Edit
So, after I wrote this, I spun up an Ubuntu server in the HP Cloud, and started playing with it. Very interesting.

The first thing I noticed was the confidence degree that the server received what I typed. I noticed this because the mosh shows how confident it is by underlining the words as I type them. For instance, when I type

echo “Four score and seven years ago our fathers brought forth on this continent”

as I type that, the underline “chases” my cursor. When the underline disappears, the client is 100% confident that the server and the client are in the same state.

Interestingly, though, when I paste in that same string, the underline is nonexistent. At least, I don’t see it at all. I’m on a Mac, and I’m using Terminal.app (yeah yeah, I know). I’m not sure if there’s an intrinsic difference in how Terminal.app handles pastes vs keyboard inputs, but the server becomes immediately aware when I paste, but not when I type. Odd.

Also interesting is the output of ps:

10512 ? S 0:00 mosh-server new -s -c 256
10513 pts/1 Ss+ 0:00 \_ -/bin/bash
10614 pts/1 S+ 0:00 \_ screen -T screen-256color ...
10633 ? Ss 0:00 \_ SCREEN -T screen-256color ...
10708 pts/2 Ss 0:00 \_ /bin/bash
10870 pts/2 R+ 0:00 \_ ps axf


All of the screen stuff is normal Ubuntu 10.10 cruft, but you can see that even though it does use the original ssh control channel to start up the connection, it drops it immediately.

I’ll run packet captures if anyone is curious, but if you’re that curious, you should just play with it yourself! Just remember to open UDP ports 60000-61000 to the machine on your firewall (which I almost forgot, natch). Let me know what you think of it in the comments below!

For a long, long time, I would hear people mention multicast, and I would be like, “oh, multicast, yeah, talking to lots of hosts at one time, yeah, that’s cool…” then do my best to not join in that conversation, because I had less than nothing to add. In fact, all I could do was show my ignorance, and THAT’S no fun.

So my shame of not knowing it drove me deeper into my cave. “It’s not like anyone even uses it. I’ve been doing networking for years and its never come up”.

Then I started learning more about IPv6, and multicast would creep into the conversation. And I’d ignore it, and sure enough, the next paragraph wouldn’t mention it anymore, and I could go back to pretending it didn’t exist. But it kept coming up, and one day I slipped.

I thought to myself, “so, I know that multicast is all about sending data to lots of hosts at once, but how could it really do that?” And so I slipped into the inescapable void of my curiosity.

As it turns out, it’s really not all that crazy, and it’s really not that difficult, at least on the surface. There are two sides of the proverbial coin. There’s the theory of how it works, and there’s the practical application. Lets look at the practical side.

A prototypical example use-case would be video streaming. Every multicast application (and by application, here, I mean every solution, such as a NASA video stream and the like) picks or is assigned a particular multicast IP address, then everyone on the internet that wants to watch the stream joins that multicast IP address…and when I say it joins that multicast IP address, I mean that the machine uses a protocol called IGMP to tell its local router that it wants in on the action.

The local router is hopefully configured to know about IGMP so it listens. If it’s configured properly, then it uses a protocol called PIM (or one much like it) to let the next closest router to the video source know that it has someone who wants to watch. That router then sends PIM updates to the router it thinks is closer to the source than it, and so on.

Eventually, you wind up with a tree starting with the root being the video server, and forming the branches are all of the routers who have clients that used IGMP to let them know that they wanted to watch. The video source sends out one packet, to its closest router, and the router remembers which connections joined that group and sends the packets to those routers, which send the packets to the next routers, and the packets eventually wind up at the clients who are watching the stream.

No one machine sends a ton of packets, and no routers have to remember much besides which interfaces belong to which groups. It works out nicely, even though, and I say this with the greatest apology now that you’ve read the preceding paragraphs, almost no one uses it like this.

Sure, there are some internal networks that use it for things like teleconferencing, but I bet yours doesn’t. Except you Googlers and Yahoos out there. Your university probably does, I guess, especially if it’s on that whole Internet2 thing, but the vast majority of you will never have to deal with multicast video streams.

That doesn’t mean that multicast is going away. Actually, with IPv6, it’s getting more airtime.

The biggest difference between IPv4 and IPv6, in my opinion, is the idea that your interface isn’t going to belong to only one layer 3 (IP) network.

You know how, right now, you are probably on a private block, like 192.168.whatever? Well, with IPv6, you’re still going to have a private block, only it’s called link-local. Your machine, and every other machine, will have a link-local network address on their interface. It doesn’t route to the internet at all. When you have one machine that wants to talk to another local machine, it’s going to happen over the link-local interface.

Of course, your computer will also want to talk to the internet, probably, so you’re going to have an IP with the “real” network prefix (which is probably going to be assigned automatically by the IPv6 autonegotiation process, which I’ll write more about later). This “real” network IP will allow you to communicate with the outside world (and it with you, unless you set up your firewall rules correctly). That’s two IP addresses so far. How many more?

Well, there is also going to be the “All Nodes” multicast group, FF01::1 for “node-local” (AKA “all interfaces on the same machine”) and FF02::1 for the link-local (AKA “the local broadcast domain” because routers don’t forward link-local packets) group. Your machine will belong to these multicast groups, because things like ARP are going away with IPv6.

Instead of ARP (and ping and passive IP collision detection), IPv6 includes a protocol called NDP, which provides for all kinds of things…and most of those features come from using multicast addresses.

So what happens is really just a smaller version of above, where the host tells the router that it wants to join a multicast group. Except that in IPv6, we don’t use IGMP anymore, we use MLD, but it works mostly the same way. The router sends us any packets that are sent to that IP address, and things go swimmingly.

If the machine that you’re working on IS the router, well, then it’s going to join any number of IPv6 multicasts…the “All Routers” multicast (ff02::2), maybe the All DHCP Servers group (ff02::1:2), and seriously, a ton of other multicast groups.

The big reason for all of these groups is service discovery. It’s kind of like, in IPv4, if you had a specific private IP block for, say, DNS servers. If you just knew that somewhere in 192.168.32.0/24, there would be a DNS server, you could join that network, ping the broadcast, and the DNS server would respond. You could use that to add it to your configuration. It’s kind of like that, but instead of a broadcast, it’s multicast.

If you’re the rare networking wonk who didn’t know multicast, you’ve probably already asked, “yeah, but how does that work at the switch?”, in which case I invite you to read the wiki entry on IP Multicast: Layer 2 Delivery.

So despite my earlier fears, multicast doesn’t operate by magic, just by protocols that I’ve never configured because they didn’t matter to me. But even with the scant knowledge that I just imparted, you can feel a little bit better about needing to use multicast all over the place with IPv6.

Apparently, USENIX hosts the Computer Failure Data Repository, which is a series of datasets from various installations that includes failure data for various components.

There are around a dozen different data sets, most from HPC clusters, but they’re actively soliciting for additional data:

If you already have your data public on your reference web page so that any one can download it, then all you need to do is to send us a pointer to your reference web page and a brief description of the data.

But otherwise – if you want to make the first release of your data through the CFDR – then the data contribution procedure is as follows:

We need to have a necessary paperwork on file to show that we actually have permission to host this data. You need to sign or find someone at your organization to sign our contributor’s agreement.
If the data contains some sensitive information like user or vendor names, you need to sanitize (anonymize) it. If you don’t have proper sanitization tools, we will try to help you.
Please provide any available documentation or description of the data you are contributing. If no documentation is readily available, it would be helpful to create one in the form of a FAQ with answers to frequently asked questions on the data. You can take a look at the FAQ accompanying the LANL data sets to get an idea of the kind of questions people commonly ask about failure data.
Make your data accessible for us, then we will host it on the CFDR server.

I’m looking forward to going through some of these data sets, just to see what kinds of things I can glean. The idea kind of reminds me of the Public Datasets that Amazon hosts, but more specific to my interests.

It’s not hard to see how crunching these kinds of numbers to get interesting statistics could be an interesting passtime (or maybe I’m just a giant geek).

Do you know of any other resources for long-term statistics on failure data related to IT? What else have I been missing?

On occasion, though, I do get some downtime to discuss things with my readers, and the other day, I got a good question that I don’t think I’ve written about, so I thought I’d share with everyone else.

Here’s the email:

Matt,

This might sound like a nooby question, but I will sally forth…

What is the difference between using a Private cloud solution and a Virtual Machine environment?

I am planning to update my test environment, and if possible use a personal Could based solution for testing various software platforms. Based on my limited knowledge of Cloud based platforms, The VM and Cloud based environments are pretty much the same, except when you shutdown an instance of cloud based machine, it loses it configuration.

I have a VM environment right now, but would like to be able to provision new machines at will (depending on hardware restrictions), but would like to keep some of these machines as configured.

Thanks in advance

Here was my answer:

Hi!

I don’t think it’s a “nooby” question. It’s one that a lot of people struggle with, because there isn’t a clear definition, and the marketing in the industry tries to confuse the matter at every turn.

I’ve thought about it pretty extensively over the past couple of years, and I actually have a definition in my mind of “Cloud” that I think is logical and based on the origination of the term. Remember that the word cloud came from network diagram icons that indicated, for the purposes of that particular diagram, it didn’t matter what specific machines or software filled that particular space, only that there was a resource available that provided a service. At its core, it is a term of abstraction.

In my mind, cloud is still a relative term. Amazon’s AWS cloud is absolutely not a ‘cloud’ to the people who work on the discrete components of it. It *IS* a cloud to you and I, though, because when we interface with it, we see no distinction between the particular constituant machines. We only see the interface to the API that controls the creation and management of the resources that are hosted within the cloud. We don’t know which actual machine or machines inside the Amazon cloud are handling our resources, and we don’t care.

It should also be noted that there are different *types* of Cloud, too, and it all depends on what you want to abstract. If you want to create entire machines and network devices, there is Infrastructure as a Service (IaaS). Sometimes, you just want to write software and not worry about administering the web server that it runs on…then you want a Platform as a Service (PaaS). It’s also possible to just want to rent a service, like hosted Email, in which case you want Software as a Service (SaaS). But since you mentioned building virtual labs, IaaS is what we’re talking about here.

So, this brings us to your question about a private, internal cloud. It IS possible to build an internal cloud such that you can interface with it in an abstracted way, never knowing or caring (other than for debugging purposes, since you’re not only running things on the cloud, you’re running the cloud itself) where the resources are hosted. But for the immediate future, its almost always overkill in a smaller shop.

The Eucalyptus Project is devoted to creating private cloud platforms that can be administered using an abstracted API – in fact, it’s the basis for the new HP Cloud that’s currently in beta.

That being said, is it worth setting up for a lab environment? I guess the answer is, “It depends”. It would be an interesting exercise to build a cloud, just for the sake of doing it, but only you know if you have the number of machines available with the (admittedly low) required resources.

It may be possible to configure your current virtual environment to deploy new images quickly and without much manual intervention, but without knowing more details, I can’t be sure of the implementation. I would, at the very least, research the API for your virtualization platform and check out something like Cobbler.

I hope this answered any questions you have. Please feel free to follow up if you have any other questions. I’m happy to help.

Thanks!

–Matt

Confusion and misconceptions about what a “cloud” is run rampant because, for a very long time, it was essentially an undefined marketing term. As the idea gels, we get a better idea of what to expect from the service providers.

Also, if you’ll note that I said that running a private internal cloud is overkill for the immediate future. That’s because I suspect more and more software will be developed with the inherent idea of “cloudiness” built-in. I’d be surprised if software like Tomcat didn’t have server abstraction built in during the next couple of releases, where you install Tomcat on a few machines, and then you deploy your application to a central controller, which then deals with the “cloud” instances. There are already commercial services for this, but there’s no reason it shouldn’t be added into the software itself.

Likewise with an abstraction layer above KVM, so you just launch new VMs onto an array of hardware, and you don’t know or care which machine it runs on. This is what Eucalyptus does, of course, but right now, Eucalyptus is exceptional for doing so, where as in a few years, it’s going to be par for the course and built in. More and more administration will be done through APIs, I’m telling you…

Of course, I could be wrong! Let me know what you think in the comments below.

There are tons of tutorials on all kinds of sysadmin-related topics, and if you couldn’t get into the tutorial you wanted, there are a whole lot of talks, too. Just check out the schedule. It’s impressive.

If you read my blog on its website, Standalone-SysAdmin.com, you’ll also notice in the upper right of the page, there’s also the PICC’12 ad. If you’re on the other side of the country, then you should register for PICC (early-bird is over at the end of the month, so do it soon!).

I should also note that both of these conferences are operated by their LOPSA Local groups. If you’re a member of LOPSA, but there’s no chapter near you, maybe you should consider starting one. The process is simple, and I’d be glad to give you a hand. LOPSA is also working on a handbook for new chapters to help kickstart the process, so if you’re interested, drop a comment on this blog entry and I’ll get you in touch with the right people.

And here’s where I make some people mad…

Major League Baseball. The National Football League. Major League Soccer. English Premier League. The National Hockey League. The National League of Nurses. Commercial Law League Of America. The League of American Bicyclists. The League of Cities. Child Welfare League of America. League of American Orchestras.

What do all of these organizations have in common? They’re all active organizations that are relevant to the 21st century and even though they’ve all got the word League in their name, none of them take crap from anyone about it.

But every once in a while, I’ll talk to a system administrator about the League of Professional System Administrators, and they’ll snicker. <voice=”fake incredulous”>League? Do you guys wear spandex, too?</voice>

Why? Because of this:

(that's me in the middle)

What actually does matter is that some people are so hung up on a name that they refuse to join the organization. This is stupid.

Granted, there are some organizations that I wouldn’t join because of the name. Lots of them start with “Nazi party of…”; the point is that while there are good reasons for not joining a group because of the name, none of those reasons apply to LOPSA.

I could write a long paragraph about how League and has been on the rise in the 21st century, but I won’t. There’s no reason to. League is fine. The dictionary definitions are all in agreement and NONE of them mention its use being deprecated.

Of course, some people claim that it’s not that the word League is so bad, it’s just that they don’t like the name. Most of the time I hear this, it’s after I shoot down their above-mentioned theories on SysAdmins in Spandex (which is another group I wouldn’t join, but not because of the name). You know what, if you just don’t like the name, there aren’t any arguments from me that can change your mind…

…but to those people, let me impress upon you a thought. LOPSA is an organization of professionals. It has dedicated itself to improving the profession of which you are a practitioner. Whether or not you join them, they’re going to continue to work on your behalf. If you agree with the goals of the organizations, but would let your personal preferences about the name of the group stop you from joining, then I can only leave you with the immortal words of my little sister:

When logging in, you get an interface similar to this:

There are three offerings immediately available. The two cloud compute options at az-1.region-a.geo-1 and az-2.region-a.geo-1, and a cloud object store at region-a.geo-1. I’m certain that they will offer both more geographic areas and services before the end of the beta, and definitely when the service goes public.

Anyway, I wanted to set up a machine and see what the process was like. I clicked on the blue “Activate Now” button in the picture above, and it took me to this screen:

The options for the size are currently as shown below:

The images available right now are here:

Since AMI on Amazon means “Amazon Machine Image”, I’m not sure what the AMI prefix means here, but whatever. I selected the smallest size and the CentOS 6.2 image with the default security group. At first, I didn’t get a chance to select a key pair, but I left instances at 1 and clicked Create. Of course, you have to have a keypair, or you can’t SSH in, so I got a warning box telling me to click “Key Pair” to make a new keypair. Done. I copied the resulting private key into my HPtestCon1.pem file, chmodded it to 400, and went on my merry way.

Incidentally, key pairs on HP are geographically specific. In running through this process again as I write this entry, I’m doing it on the az-2.region-a.geo-1 compute group. I don’t know if this is going to remain the case for public production, but I hope not. I would like my in-production servers to not need me to be aware of which geographic zone they’re running in. I don’t know if AWS or RackSpace are set up in the same way…maybe someone can chime in below in the comments?

I asked @HPCloud on twitter, and I got a response from HP cloud guy @rupakg:

When I asked about the ability to manually add keys (because I don’t really want to have my scripts care which geographic zone these VMs are running in), he suggested that I check out the Cloud CLI. I haven’t had a chance to play with it yet, but it seems promising. There are Powershell commandlets for Windows and a Ruby gem for Unix, in addition to Eucalyptus tools, which makes sense, since all of HP Cloud is based on OpenStack.

After creating a key pair, I was able to start the new instance with no problems whatsoever. Of course, once I started the instance, I ran into a small hiccup…my Private IP was 10.4.7.whatever and I didn’t have a public IP to connect to it using. Right. So I clicked on the instance number listed, and one of the options was “ATTACH PUBLIC IP”, and all of a sudden, I had a public IP. An IPv4 public IP.

I’ve been looking for a cloud provider that has IPv6 out of the box, and sadly, HP doesn’t have it. That’s ok, as far as I can tell, no one else does, either. [Edit: Several people have told me on twitter that Linode has it. Yay!] I still wanted to set up an IPv6 web server as a demo, though, and I wasn’t going to let this stop me, so I went through the path of least resistance: http://www.tunnelbroker.net/.

I created a TunnelBroker account in about a minute. After logging in, I clicked “Create Regular Tunnel”, and was asked for my IPv4 Endpoint, and I gave it the IPv4 address that was assigned to my VM. It then checked the address, determined that it was a potential tunnel endpoint, and suggested I use the Chicago tunnel server. Sure, why not? I clicked Create Tunnel, and in a few seconds, it took me to the Tunnel Details page:

Easy peasy. Anyway, my server was up and running, so to connect to it, I just ran

ssh -i ~/HPtestCon1.pem root@mypublicIP

Once on the server, I quickly configured the IPv6 tunnel from Hurricane Electric. In the Tunnel Details above, you can see the “Example Configurations” tab, which I clicked. Once on that page, there’s a pulldown for “Select your OS”, and when I selected Linux-net-tools, I got the following:

ifconfig sit0 up
ifconfig sit0 inet6 tunnel ::209.51.181.2
ifconfig sit1 up
ifconfig sit1 inet6 add 2001:470:1f10:294::2/64
route -A inet6 add ::/0 dev sit1

Pasting this into the shell didn’t produce any errors whatsoever, and I tested my IPv6 connectivity:

[root@server-49193 ~]# ping6 ipv6.google.com
PING ipv6.google.com(mil01s17-in-x10.1e100.net) 56 data bytes
64 bytes from mil01s17-in-x10.1e100.net: icmp_seq=0 ttl=54 time=185 ms
64 bytes from mil01s17-in-x10.1e100.net: icmp_seq=1 ttl=54 time=185 ms
64 bytes from mil01s17-in-x10.1e100.net: icmp_seq=2 ttl=54 time=185 ms
64 bytes from mil01s17-in-x10.1e100.net: icmp_seq=3 ttl=54 time=185 ms

— ipv6.google.com ping statistics —
5 packets transmitted, 4 received, 20% packet loss, time 4016ms
rtt min/avg/max/mdev = 185.038/185.082/185.142/0.432 ms, pipe 2

Excellent. Now for Apache. First, lets install it:

yum install httpd -y

Now we have to configure it. Edit /etc/httpd/conf/httpd.conf, and look for the Listen lines which dictate which IPs the machine responds to. The default line number is 134, if you were curious.

Because we want to be specific, change the line that says:

Listen 80

and change it to

Listen 10.6.whatever.whatever:80

where 10.6.whatever.whatever is your private IP.

We’re also going to add a line specific to the IPv6 address that we’ve been given by Hurricane Electric, so in the commands above, I would use 2001:470:1f10:294::2. That’s my side of the tunnel that they set up for me. Yours will be slightly different. The line I added was:

Listen [2001:470:1f10:294::2]:80

You’ll notice that there are square brackets surrounding the IPv6 address. These are very important. The reason is that, historically, the colon separated the IP address from the port number, as it does in the IPv4 line above. With IPv6, we can shorten the IP by replacing the longest string of zeros with :: – thus making an IP of 2001:470:1f10:294::2 when the full length address is 2001:1f10:2940:0000:0000:0000:0000:0002. Having colons define a port number doesn’t work anymore unless you do something else to determine the IP address, so the standard operation is to enclose it in square brackets. Make sure to do that.

Save the file, then start up Apache:

[root@server-49193 ~]# service httpd start
Starting httpd:
[root@server-49193 ~]# service httpd status
httpd (pid 3002) is running…
[root@server-49193 ~]#

Now, we can test it by going back to the Hurricane Electric page, and clicking “IPv6 Port scan” under “User Functions” on the left. It asks for the IPv6 address to check, so put your end of the tunnel (in my case, 2001:470:1f10:294::2). I put it in, and the submit button didn’t light up, so I hit tab, which made it start checking the IP. The IP was valid, and I could click submit, so I did.

The output on the Nmap window came back like this:

Starting Nmap 5.00 ( http://nmap.org ) at 2012-03-11 19:17 PDT
Interesting ports on standaloneSA-2-pt.tunnel.tserv9.chi1.ipv6.he.net (2001:470:1f10:294::2):
Not shown: 998 closed ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http

Nmap done: 1 IP address (1 host up) scanned in 17.62 seconds

Exactly what I was looking for!

So there you have it. Around 5 minutes to set up IPv6 on a web server, and a quick walkthrough of the new HP Cloud beta. I think it has a lot of potential, and I’m looking forward to seeing where HP takes it. You can watch as news comes out by checking the HP Cloud Blog. They’re also at SXSW, which I really want to go to next year. I had a really great time in Austin for Tech Field Day 7…I can’t imagine the spectacle that must be South by SouthWest.

Anyway, I hope you enjoyed this walkthrough. Let me know if you have any questions about this that I can answer (or find you the answer to)!

In it, I discussed how system administration would become development, much more like coding than like swapping disks…

The future is already here — it’s just not very evenly distributed.

–William Gibson

Basically, yes, it has come to pass. No, not everyone is using that particular strategy, but the number of people who aren’t will continue to get fewer and fewer, especially as cloud-based offerings get more and more prevalent, accepted, and expected. But all of this is just tangential to what I really want to talk about.

System Administration has to divide in order to continue to grow. Tom Limoncelli has been saying it for a few years, but I did’t really get it until relatively recently. I don’t know if we agree about the underlying reasons, but I suspect we’re on the same page. Let me lay the ground work, and then I think you’ll also agree that it’s pretty much inevitable.

System Administration is a relatively recent field compared to pretty much every other professional endeavor. Medicine has been around in an organized form for over 100 years. Accounting for 400-500 years, and other..um..professions…for considerably longer. (Note: this is not to say that the art and science of these are not much, much older, but I am speaking here of organized professions).

The profession of System Administration is still coalescing, but that’s only because we’re still very much in the formative years of the art and science. It’s exciting, because we get to experience the transition from humors and leeches into germ theory and beyond.

As we’ve progressed, something has become clear; a single person is unable to handle all aspects of the job. We’ve all seen this, of course. There are administrators specific to networks, storage, virtualization, applications, and more. This division has been caused by the domain-specific knowledge required to adequately perform a certain aspect of running IT. Not everyone can know everything…

It is this last point that is going to cause the division that I speak of. Currently, the field is very roughly divided into junior and senior sysadmins. The Job Descriptions for System Administrators short booklet by USENIX is a good effort to add some qualification to the distinctions between the various job levels, but it still supposes a continual advancement from the very entry level administrator up through senior, level IV System Administrator. I don’t think this can continue for much longer, particularly in an infrastructure of any complexity.

As time progresses, here’s what I think will need to happen (and in some cases, it’s already happening, but it’s going to become more widespread).

We’re going to start seeing two career paths for people who work with servers. At the top of the professional career paths, there will be the people who are in charge of physical infrastructures, and there will be people who are in charge of the operational infrastructures. The inevitable move to cloud-like infrastructures is going to mean that, to the operational architect, the physical hardware is almost completely irrelevant. At the same time, the physical administrator is going to be concerned with those aspects of running the infrastructure, such as physical Moves, Adds, and Changes (MACs) as hardware becomes more numerous and anonymous.

Almost no one is the size of Google, so when machines die, they will need to be replaced. They will need to be monitored, and their performance, independent of the software installed on them, will need to be monitored for efficiency, with underperforming equipment being fixed. The cloud layer of abstraction will allow machines to be pulled in and out of service to be repaired and replaced, but certainly a degree of expertise will required to engineer the best, most efficient datacenter solutions. This is what the top tier physical infrastructure will concern themselves with.

The other side of the coin is the operational administrator, who concerns themselves primarily with ensuring the business operations and continuity. If the physical infrastructure admin’s concerns are hardware, the operational administrator’s concerns are data and software. The operational admin won’t ever have to consider what disk array to purchase, or how large the storage needs to be, but will instead concern themselves with things like the performance growth rate and the number of CPU units required for a particular task. Infrastructures will be developed like software is now, using the same methodologies, sharing some of the tools, if not the languages. Someone in a senior operational role will need business intelligence, as well, because they will be responsible for translating business directives into infrastructure.

These specialized roles will have junior members, of course. An entry level physical infrastructure admin will probably enter the field in the same way that many of us did, as an apprentice, then moving to a journeyman type of level while learning more. I imagine that more education will be necessary to fill the truly senior physical roles, but it’s not inconceivable to learn everything on the job.

In the operational role, it’s harder to imagine many people making the same transition from non-admin to admin. My own path, from Tech Support to SysAdmin, won’t be nearly as common. Many current admins were able to make the transition between non-admin and admin because the operating environments are similar on their laptops and their production infrastructure. Once the operating system ceases to be important, this easy path will disappear, and the only people who make that transition are those who actively work at it.

Because the operational aspects have such an emphasis on programming, it’s easy to see that more administrators will become sysadmins directly out of CS programs at universities. Hopefully the people who dictate the curriculum will take notice of this and start adding operational electives.

I don’t have a time frame for this, but I suspect that in another four years, we’ll see more and more companies coming online with this model. Over the past couple of years, I’ve seen a lot of blog entries and online articles talking about the disappearance of System Administrators. These articles are mostly hyperbole. The sysadmin of today is not going to disappear, but they are going to be fewer in number.

Companies who, right now, have their own physical infrastructure are not going to go through a lot of expense to get rid of it, but companies who come online in the future are going to see more and more reason to use a public cloud service, which means that they’ll only be employing the operational administrators. Sure, there will still be desktop support in most cases, but in terms of sysadmins, for those companies, it’s mostly going to be architects. Once a company grows to the point where it’s not economical to host at Amazon, they may build their own cloud infrastructure, in which case they’ll employ the physical infrastructure admins, too.

Just so you know, this isn’t a dystopian future. Specialization is a good thing. It’s going to require us all to work together, but the end result will be that we can accomplish much more than we could before. The knowledge level required to do our jobs is going to go up, but we should see this as a challenge to learn more and to be better.

I guess I never really understood how important it was to have a pile of legacy kit sitting around waiting on me to hook it together (or, more likely, tell my junior admin to hook it together for when I had time to play). When I missed my whiteboard, I went to Home Depot and got a bunch of dry erase panel and covered the garage in it. My wife thought that was weird, but she basically understood. I’m not sure she’d be so tolerant if I asked for a rack and some servers to fill it.

For this reason, I’ve been investigating what my options are. There are some easy ones, of course. I’ve got a newish unibody MacBook Pro, so VirtualBox runs great on it, especially with 8 GB of RAM. You can technically upgrade it to 16GB, but that’s really expensive, so I’m not going to do that just yet. Besides, at this point, my laptop is very I/O bound.

Incidentally, you CAN run ESXi(vSphere) under VirtualBox, but it’s pretty much a “cause I want to practice with the interface”-only kind of thing. You can run VMs, but it’s not really pretty.

So although I can do some simple prototyping in VirtualBox, in order to do some heavier lifting, I’ve got to figure something else out. I actually have two pieces of physical server hardware that I picked up from a friend. One is an old Dell machine that is technically 64 bit, but was made before the VT-x extensions. The other is a 32bit HP Proliant ML570 7u beast. It’s got some possibilities in terms of disk space, since there are two 6-disk arrays in it, but it’s huge and loud and expensive in terms of electricity usage. (By the way, if you’re interested in either of these, just say the word and name your price).

Honestly, I’ve been thinking about picking up a couple of these PowerEdge 1950s. They’re decently upgradable, they’re 64 bit with VT-x, they’re on the ESXi HCL, and two of them are still less than half of the space taken up by the HP in my garage.

I’m looking into the cheapest way to get some kind of shared storage solution, which will probably end up being one of the BSD appliances that run ZFS on a bunch of hard disks (something like NexentaStore, maybe). The real question is what I’m going to put the hard disks in. The ML570 is just a bit long in the tooth for this kind of thing, and as I said, really inefficient. Something like this might be nice and unobtrusive.

It's dangerous to go alone. Here, take this cloud security blanket.

One of the things I’ve been doing in the past week or so is to start investigating Amazon’s pay-to-play Amazon Web Services. It took a while to set up the environment on my laptop that I use to administer the machines remotely, and it took a little longer to really kind of figure out the Amazon mindset, but I think I’m actually starting to get the hang of it, and I haven’t even cost myself more than $5 yet.

The thing about Amazon that is kind of scary is that you really are charged per hour. What this means is that I don’t run my Amazon lab for long periods of time. In fact, I’m kind of paranoid about it, so every once in a while, I’ll run ec2-describe-instances just to make sure I’m not accumulating machine charges. I’ve made my instances all of the micro variety, since that’s only $0.03 / hour that the instance exists (note: it’s not enough to shut down the machine; that doesn’t kill the instance. You need to terminate the instance, which also erases the ephemeral storage.

I would be shirking my responsibility if I didn’t also mention that in the spirit of drug dealers everywhere, Amazon is offering a “first one’s free” policy, which gives you a 7500 hours worth of free micro instances. So check that out.

If you have any interest in a “dipping your toes in the AWS water” type post, comment below and let me know. I’m starting to dig the AWS thing.

I haven’t investigated the Rackspace cloud much. The prices seem alright (actually, the tiniest servers on Rackspace are only $0.015 / hour. I’m not sure how much you can do with 256MB of RAM for an hour, but it’s great for testing implementations, which is really all I’m doing.

I’ll probably start playing with the Rackspace stuff eventually, too. There just aren’t a ton of options in this arena to play with. Oracle has a cloud offering, but it all seems to be Oracle as a Service (or OaaS, something I’ve said phonetically for YEARS when talking about Oracle). If someone from HP wanted to invite me into their cloud beta offering, I wouldn’t complain, either.

[EDIT] As it turns out, I got into the HP cloud beta. Sweetness! Thanks HP! I wrote a walkthrough on setting up an HP cloud instance and getting IPv6 up.

Anyway, that’s where I stand. I’m itching to get a more permanent lab environment running here, but in the meantime, I’m relying on the cloud providers to get my technology fix. Anyone have any suggestions or want to talk about the cool stuff they’re doing with their lab so I can be jealous?

I’ve gotta tell you, I’m really excited about this, and not just because I’m a big Mac fanboy. And you should be too.

Have you been monitor shopping lately? Yes? Then you know how much it sucks. Why? Because of the curse of 1080p.

A while back, someone decided that the “right” resolution for TVs was 1920×1080, and the best way to draw the picture was progressive scan (as opposed to interlaced), so 1080p became the prevailing TV resolution. Which was OK, but still not great. Somewhere along the way, though, things went off the rails…

Companies making monitors apparently decided, somewhere in the 2005-2010 range, that everyone wanted to watch movies on their computers, which is a logical statement. I don’t only want to watch movies on my computer, but I would like to do that. And apparently the monitor makers thought that’s ALL I would like to do, because over the past few years, pretty much every monitor that you can buy at a decent price is 1080p.

Don’t believe me? Check out Pricewatch’s list of 27 inch LCD monitors. Lowest price is $264 as I write this, and it’s 1080p. The next nine are, too, until you get to this $900 beast, which runs 2560 X 1440. Say, that sounds pretty close to the iPad’s 2048×1536…

Yes, ok, I’ll admit it. The reason that I want the iPad to have a retina display is so that every other LCD maker in existence is shamed into making better screens. The ability to hold a $1,000 tablet that includes an 9.7″ screen that pushes nearly twice the pixels of any screen you can buy in the whole big box store down the road should be sufficient embarrassment to get them off of their butts and their laurels.

There’s absolutely no technical reason for this kind of stagnation, as Apple will hopefully show everyone. If it takes Apple to accomplish what NEC or ViewSonic should have been doing years ago, then so be it, but we need better displays, both in terms of resolution and acreage. There are only so many terminal windows that fit on 1080p, and I’m tired of having to add more screens to get the space I need.

There are 10 kinds of people in the world.

Those who understand binary, and those who don’t.

Yes, hahaha, very amusing. You can even buy it on a shirt. But here’s what I just realized…it doesn’t work as a spoken joke at all.

While writing about binary and decimal, I’ve had to be very careful to pay attention in my sentences to whether I’m writing the word for a value or I’m writing a number. The number 10 is very different than the value ten. The joke works with “There are 10 types….”, but you can’t say “There are ten types…”, because 10 isn’t ten in binary. In binary, 10 is two.

I may be the slowest horse here, but the clear distinction just occurred to me that a written word indicates a value. I am now retroactively aggravated at everyone I’ve ever talked to that pronounced “10″ as “ten” when they meant anything except the decimal number 10.

Am I wrong?

I’ve been arguing my case with Philip Durbin since then on Google+. I’ve had almost innumerable conversations with people on twitter about it. And it apparently spurred Wesley David into writing a piece talking about the moral side of the issue. Interesting stuff.

And for the record, because A LOT of people have asked me about it, my definition of theft is straight out of Merriam-Webster.

I think if you read the article that I wrote, I never said anything about right vs wrong, or illegal vs legal, and that was on purpose. I have mixed feelings about all of it. It’s obviously wrong to take something of someone else’s without their permission. On the other hand, I don’t think that it’s necessarily wrong to do something that’s illegal if the law is in the wrong, either.

Slightly related is YCombinator’s recent Declaration of War on Hollywood. YCombinator wants to fund startups that will eventually eliminate Hollywood Studios.

Joel Spolsky recently wrote a piece on Google+ advocating legislation that causes the elimination of software patents and the rolling back of copyright protection. And this is from a guy who’s company makes copyrighted software.

Between you and me, I think there’s something going on, society-wise. As a group, we’re looking at things differently than we did, even two years ago. I’m not sure if all of the SOPA blackouts would have happened then. I don’t know if it changed with the Arab Spring or not, but it just feels like people are more cognizant of their ability to make change to the world around them. And I think that’s a great thing, and very promising.