Post-LISA Pre-Holiday Filler

LISA was pretty exceptionally late in the year this time, being only a couple of weeks before Christmas. The inevitable let-down of coming back to the “real world” where I’ve got to “work” and “do productive things” is always kind of a drag, but this year is a little different thanks to the proximity of the upcoming holiday plus the fact that I work for an .edu now.

Here at NEU, we actually have the entirety of next week off, plus we don’t come back until Wednesday the 2nd. Like, the whole week plus! I first heard about this back in August when I started, and I was like, “oh man! That’s great! We’re going to have so much time to do infrastructure work. We can take down the whole network and no one will care”. Because, you know, I’m crazy like that. My coworkers quickly mutinied against my ideas because, well, it’s a week off. I’m beginning to see the wisdom of their mentality, and I’ve made plans to head back to Ohio to visit with family for the Christmas break.

I still do kind of wish I could spend some time in the server room fixing things up, but I’ll take care of it next year. I have some pretty large plans, and since Amy and I will be taking the train to Pittsburgh (it was actually the cheapest way to get there – I can’t believe it either), I’ll have plenty of time to make plans and write. I’m really looking forward to the trip.

Here are some of the things I’m working on:

  1. VLAN Renumbering Project
  2. Our network design is actually pretty archaic. We’ve got several networks where desktops and servers are in the same subnet (what I call a hybrid network – and warn people against). I’m going to be dividing them up. Plus, we’ve got all kinds of subnets which have the same sort of security needs, but are in separated networks for no reason that I can discern.

    I’ve worked up a security zone “map” of all of the types of access that servers need (and need to provide), and I’ll spend some time on the train figuring out what logical grouping of servers, desktops, and appliances makes the most sense. I’m sick enough that I actually kind of like stuff like this.

  3. Server Room Rebuild
  4. We have the cheapest racks known to mankind. Well, ok, second cheapest – they do have four arms. But they’re really bad. They’re round-holed, have no cable management features, and the one attribute they have that I don’t hate is…wait, no, I don’t think there’s anything I don’t hate about them.

    As part of the three year budget estimate that I submitted, I included a request for a new set of racks in the server room. I also want to change the way that the server room is laid out. At the moment, we’ve got three rows of racks, and, well, the airflow is kind of interesting:

    Old Server Room Design

    It’s not just me – that’s crazy, right?

    Anyway, I’ve done a survey, and of the 882 rack units in that room, we’re using in the neighborhood of 350. Most of everything else is taken up by free space or by shelves holding up desktop machines turned into servers, most of which we don’t want to keep around anyway. So yeah, here’s what I want to do:


    I think that makes a lot more sense. We still have over 500 rack units of space, plus we get a much better airflow with less mixing, and we can use panels to further help separate the hot and cold aisles. It should be a lot more energy efficient. Plus, I can get rid of these damned round-hole telco racks. Yech.

  5. Network Core Upgrade
  6. Right now, our “network core” is a Cisco 6509. Not a 6509E, mind you, but an old-school 6500-series that has been EOL’d (stage 6 in that document). I see this as an opportunity.

    Not only is our 6509 our core, it’s also, in a large way, our distribution switch. Well, one of them, anyway. It’s stacked full of 48 port gigabit blades (including a couple of really crappy cards that don’t even support the crossbar). I want to fix this.

    My thought is that it makes a lot of sense to replace our one 6509 with two 6506Es, and use a “top of rack” (ToR) switch network design where we actually have a ToR switch every other rack or so, then wire every ToR switch to both 6506Es for failover. The number of ports we’ll have at our disposal is higher, we’ll have a more robust design, and unicorns will pop into existence ready to cater to our every whim.

  7. pfsense
  8. Related to the weird networking layout, we’ve got an array of firewall boxes, some pfsense, some bluesocket. Apparently, the bluesocket machines fail pretty frequently. I haven’t seen it yet, but I believe them. The replacement of the existing array into one or two (clustered, of course).

So that’s what I’m working on. I’d also like to get further into coding some scripts for AWS. I found out about a great python library called boto. It’s completely full-featured. The only drawback is that it’s written in python ;-)

As a side effect of doing some preliminary coding with boto, I’ve been working on IDEifying my vim. Those of you who use vim (and you should) may want to check into these plugins if you aren’t already: NERDTree, snipmate, and vim-surround. I’ll be doing a vim-specific post at some point, so if you have any awesome plugins, let me know in the comments.

That’s it for now. It’s a short week, so back to work!

  • John M

    We just finished a core upgrade (one 4507 for two 6506’s… Don’t ask), and instead of ToR’s we used patch panels at the ToR for redundancy. Cheaper than dual switches on the racks, and the wiring was less of a mess at the switches.

  • Hi John!

    Thanks for the comment. I had thought about doing that, but I’d still have so much vertical wiring going back to the central switches, plus for failover, I’d have to run each machine to two switches – that’s a lot of network ports! (unless there’s a trick I’m missing? Please let me know if so!)

    But yes, definitely cheaper than dual switches :-D

  • John M

    Since most of the servers had 2+ network ports, we just bit the bullet and ran cabling to two patch panels above the core switch stack, and patched the panels from there to the switches for redundancy.

    We required the redundancy for a our VM infrastructure, and our file server cluster, so the decision to create redundancy for all the racks was easier and less costly that dual ToR switches and fiber interconnects to the cores.

    A lot of ports? Yes.

  • Interested to hear why you do not like VLANs with a mix of servers and workstations. Often a setup like this reflects the actual geo-political or business case (e.g. Finance department computers and Finance related servers segregated, etc.). It also seems to reflect well against purpose built VMs that are serving a designated purpose (as opposed to the old school physical box that had to run everything). Many ways to skin a cat of course, but this approach surfaced for a reason.

  • John M

    @Pete: The idea of using VLAN’s to segregate servers from desktops is for security and manageability.

    You don’t want updates for desktops on the same network as those for servers, since servers (most businesses, anyway) don’t update servers as often as desktops. This traffic will impact services that are delivered to desktops, causing timeouts and delays in network response time. If a virus get loose on a desktop VLAN, it is easier to compartmentalize the damage, and block (hopefully) them from affecting your servers.

    As for management, you don’t want to send management traffic across the desktop network due to the possibility of interception of information that you want the servers to see due to insecure management protocols (RDP for instance, sends usernames and passwords in cleartext, still).

  • For the data center, have you considered implementing fully-enclosed cold aisles? I’ve seen this in data centers in Europe, and it increases the efficiency of cooling. Another option is fully enclosed cabinets where the cooling comes up from below. This is an even more precise way of targeting your cold air. Also, don’t forget your brush kit/grommets to seal up all the holes in the floor:

  • Matt,

    You really have physical servers with just one 1Gbps Ethernet port? and aren’t visualizing them?

    I’d consider fabric based switches rather than a pair of 6509Es. You could build a net with more bandwidth, no spanning tree delays on re-configs and a clear path to 10Gbps for less than the 6509s will cost.

    Building out my own new DC (about 1/2 the size of yours) lots of blog posts at to come.

    – Howard

  • OK, taking these comments in order:

    Pete: John hit the nail on the head. I can’t control access to the servers nearly as well over a layer 2 network as I can by segregating them across layer 3. Also, there are some servers that shouldn’t be accessible at all over the net, and I’m going to be throwing others in a DMZ. I’ll write about it when I do it, for sure.

    Brian: Someone else suggested doing the fully-enclosed thing. I’m not opposed to it, but I’ll have to check out what that entails on the physical plant side. I don’t know that our space will support it (and it definitely won’t support under-floor air supply). Thanks for the suggestion – I will be looking into it further.

    Howard: Really? I thought the Nexus lines were way more expensive. I priced out some 7000 series chassis with associated controllers and FEX and it came to a decent amount more, if I remember correctly. I’ll get in touch via email. Thanks!

  • In other news, a really useful plugin I’ve grown to love is called Reimin. It’s a bit older (last update was 2+ years ago) but rock solid. When you call it, it prompts for a path, and automatically adds the appropriate include/import/require/etc line for you. It supports python, so if you ever find yourself missing an import statement, I would recommend using this plugin so you don’t have to jump up & down the file. :-0


  • Re: Fully enclosed aisles

    The ones I’ve seen were just a few extra panels across the top of the aisle and then panels on either end with doors on them. It didn’t seem to require any major physical changes to the structure of the room, as they were attached to the cabinets.

    It does, however, make panels very important in every blank U in the cabinet, as the enclosed aisle needs to be completely enclosed.

  • Re: Matt and John,

    I’ve seen topologies in which categorizing servers and workstations in their own designated VLANs served little more than giving some impression of order, rather than actually serving the security needs of the organization. Little to no ACLs because that approach didn’t suite their organizational structure, and caused more problems than it solved. This approach may not account for the true security divisions within an organization (defined by the business, etc. and not by system type). This of course is done all the time at a larger scale, when one has to consider geographical boundaries in their L3 design. The other thing that comes up is that there is some mutual exclusivity with that model. What is a workstation, and what is a server, and how is that defined? Sounds like a silly question until you look at say, a Software Development environment in which many systems simply can’t be categorized that way. I’m not suggesting that the approach you described is wrong in any way. In fact it can be, and is fine in many organizations. However, I’ve seen other implementations that were based more off of security divisions within the organization, and it was a very logical and successful approach.

  • @Matt as Howard mentioned, Nexus 5k line comes out cheaper than 6509Es. So you might want to look at that. Catch(and a big one for us): There are still lots of bugs though if you want to run these in active/active config. Config sync from cisco is a joke for Nexus line. Hopefully, cisco can get these fixed quickly.
    We ran a blog post recently on top tips for cable management. Check it out here:

  • Your proposed new rack layout would actually lend itself to hot aisle containment rather than cold. The cheapest, simplest implementation I’ve seen is plexiglass providing a roof over the aisle and the ends sealed with vinyl strips (e.g. Of course you also need to blank out empty RUs and ensure your racks don’t “leak” too much air in either direction.

  • manoj jaiswal

    Spaceage sell a Server Room Environment Monitoring Manufacturer, distributor, dealer, supplier, exporter in India and Delhi/NCR