Running Cables - How I do it

Date August 21, 2012

When I woke up this morning and started thinking about what I had to do today, I realized that I'd never really written about the process of running cables in the server room. Lets fix that!

So, step one is to plan what you need to do. I'd be all sarcastic, and say "obviously", but to a lot of people, it's like, "What? We don't have enough cables? RUN MOAR CABLES!". Before we go all 'Leroy Jenkins' on this, lets sit and consider the answers to these questions:

  1. Which servers need cables
  2. How many cables each server needs
  3. How long of cables do we need (aka "Where is the Machine?")
  4. Do we have enough available switch ports?
  5. Do we have enough cables of the right length?

Once we know these things, we'll be in much better shape.

In my case, these additional cable runs are a small piece of the larger goal of replacing our main NetApp filer. Some of our virtual machines had been having issues with network IO contention during our regularly scheduled snapvault jobs. Our workaround had been to throttle the process through software, but the new filer has some extra ports, and "gee-wouldn't-it-be-nice if we used those to create a dedicated channel for the snapvault process?".

While I've got my cable-running hat on, some of the VMware machines were initially configured with redundant interfaces on their management vswitch, but because we didn't have enough free ports at that time (remember to check #4 above!), each server got only a single cable. So I might as well take care of that, too.

In the case of our brand new shiny NetApp 3210 filer, there are two controllers, and each controller has 6 interfaces (since we added a 4-port card). Four of the ports are bonded together in an LACP trunk which has multiple virtual interfaces so that it can be connected to several VLANs.

That leaves two interfaces for our nefarious purposes, and since we have two controllers, we'll need 4 cables to the NetApp.

There are seven vSphere hosts, and I manually checked each one to see how the management vSwitch was configured. I found that three of them had been configured to have redundant connections, but of course, there was no cable plugged in. When I take care of that, at least the annoying error icons will go away. The other four needed me to add another interface to the vSwitch, so I did that through the interface. It worked because there were only seven. If I had 70, I'd have been writing myself some PowerShell.

So now I knew how many cables I needed. We don't yet have a rack-level map of the serer room, but we do have 21 racks…(this lack of a map will be fixed, promise), so off to the server room I went to figure out the distance.

When measuring distances for cable runs, whether in the server room or in an office space, don't forget to take into account the vertical length. Racks are 42u tall, so make sure to account for where your equipment is. Always use a longer cable than you need to account for bend radiuses and slack.

Of course, some people prefer to make their own cables to length, rather than buy pre-made patches. I understand why they do this, but I disagree that it's worth the tradeoff. Having an extra coil or two of network cable on top of the rack isn't the end of the world, but if your hand-crimped end comes loose some day, it's going to make for very intermittent issues at an inconvenient time. It doesn't matter how good you are at crimping cables, you're not as good as the machine that crimps the cables and applies the molded ends. So just buy the cable to length, if at all possible.

As it turned out, I didn't have enough cables of sufficient length to make all of the runs today, so I settled for the four most important lines (the connections to the NetApp). Which was actually pretty good, because finding open ports on our switch was...troublesome.

As an aside, right now, we don't really have a network "core" like you would think of one. We've got a big 6509 switch that everything in the server room plugs into, including the uplink to the rest of the university. Although it does work, it's definitely not optimal. The network layout that I want to go for is this one:

What this gives me is better resiliency, it has the capability of better performance, and I can expand a lot more with this. Plus if a "core" switch dies, the other one is still alive and kicking, and will take over.

But I'm not there yet. Some day. Until then, I've got to deal with a current lack of network ports...so I only ran the four. But I didn't just grab four cables and climb into the ceiling. Oh no, we've got to get them ready.

The first thing I did was to determine where on the switch I'd plug them in, and I documented which cables would be going to which ports. Then I took the cables out of the bags, left them in the coil, and used a cable tester to make sure that they were good. The chances of a machine-made cable being bad are low, but not impossible, and I'd prefer to only climb through the ceiling once a day, if possible.

After the cables tested good, I still didn't unwind them. I labeled both ends of the cables (do you hear me? BOTH ENDS) because this would be the last point where it was obvious which cable ends were matched. As soon as you unwind the cables, you have to start tracing if you want to figure out which end of the cable matches the other. Label them before they're unwound and you'll never wonder again.

Once the cables were tested and labeled, then and only then did I unwind them. Since I had to use 30' cables, I needed a place long enough to stretch, so I went out in the hallway.

It would be very good to take this time to straighten out all of the kinds and loops that are still in the cable at this point. You want it to be as easy to run as possible, and you don't want to have to deal with all kind of crazy loops at the same time you're trying to not fall off the ladder.

After getting it reasonably loop and kink free, I looped it into loose coils and trudged to the server room, where I began by evaluating how the existing cable runs to that rack were accomplished. I noticed that there was a small loop of extra cable by the NetApp filer, but that the bulk of the excess was at the switch-end on top of the racks.

I put the loose coil of cable on the ground near the rack with the filer, carried the remote end up on to the ladder with me, and started feeding it across the cable ladder in the direction it was headed.

I feel like I could have done a better job of keeping the bundle together during this time, but I'm honestly not sure what I could have done. I used some twist ties (4 or 5 of them) along the bundle to make sure that the individual wires didn't get too far out of control, but I still had to deal with a lot of loose wires. I'd be interested in hearing from someone else what they do, and what a better way is, if I'm doing it wrong.

Anyway, I got the bundle up onto the cable ladder going parallel with the existing bundles that were up there. Once a section was in place, I took apart the velcro wrap and put included my new bundle with the old. Here's a picture of the middle of the process:

The four darker purple lines heading straight down in the middle are the ones I was working with. You can see I've got them fed through the brackets (which are overflowing - something else that will be fixed with top-of-rack switches), and I'm getting ready to bundle them into the rest going to the switch which lives in the rack directly underneath the cable waterfall there.

At this point, it's simply a matter of configuring the switch ports with the right settings and we're done.

Cable running isn't necessarily fun, but it doesn't have to be terrible, and there are things you can do to make it easier. The biggest one is to just plan ahead, though. I'm very much open to what you think of my process, especially if you have any pointers on how I can improve it. Just comment below to tell me. Thanks!

  • Pingback: Running Cables – How I do it « Storage CH Blog()

  • http://www.rootmytoaster.org Kenny

    I'm going to be honest, I just imagined you running between some server racks, holding hand fulls of cables yelling "LEEROOOOOOOY JENKINNNNNNNS!" At the top of your lungs.

  • John

    I noticed that you didn't mention what kind of documentation that you put on the cable labels, or the bundles.

    Do you date the runs of new cables, and keep a spreadsheet (etc.) of the runs/bundles of runs for troubleshooting?

    I am involved right now with phone system migration in which I had to remove some network equipment, and I took the time to document the topology. Granted, it is a small IT rack, but it wasn't done.

  • http://deranfangvomende.wordpress.com darkfader

    Hi,

    I'm quoting my very favorite part from "SAN Lessons learned". The golden line is the one saying "excess cable must be at the host end"

    "Cable Management

    Cable management is an important issue and necessary to address during setup. Neatness is of utmost
    importance. If a cable cannot be traced or new cables can not be run in a timely manner, it is nearly
    impossible to replace failed equipment without affecting other users and can make troubleshooting very
    difficult. A label with start and end points at both ends of the cable may seem redundant, but it is helpful
    when running multiple cables at once as well as when replacing failed hardware to keep the cables on the
    same port. Keeping cables on the same port is important for troubleshooting and maintenance and
    obviously necessary if port-based zoning is in place. Coiling excess cable at the host, where there is more
    room will reduce clutter at the switch. Even in a full size rack, there will be only forty host connections
    (eighty with dual HBAs) compared to 256+ in a switch rack."

  • http://deranfangvomende.wordpress.com darkfader

    Ah, and one thing I tried in our new office was 1:1 wiring the switch ports to the patch panels. So everything is managed from the switch itself, no cables are re-wired.

    Ideas from the dreamland side of things:
    - Write a module for our switches to drive config from RackTables. This has been written in PHP and anyway, I can't code. But it would be neat.

    - Autolocate NagVis service icons on the rack photo by using:
    defined trunks of cables for each height unit in a rack.
    The LACP Port group Ids, or macaddresses and the known cable trunks.
    "Ask the switch where your server is"

    Oh and my biggest dream would be GVRP including VLAN names. :)

  • Pingback: Sysadmin Sunday 94 | Server Density BlogServer Density Blog()

  • http://allfreeinstagramfollowers.blogspot.com how to get more followers on instagram

    In this Tutorial I will be explaining to you how I get my Intagram Followers!
    Instagram has over a hundred Million users, which is clearly a whole lot!
    Now don't you want that a few 1000's of the 100+ Million end users would follow you??
    It can not be that hard to attain this thinking about the mass sum of customers that use Instagram.

    That's why I am training you all how to get limitless
    Instagram followers! I have produced this distinctive tool which will permit you to obtain a mass amount of
    followers in a short amount of time.

    I will be supplying this to the group out there for everyone to use!!
    To get your Instagram Followers, you will need to have to down load
    the system from the url under (virus scan is presented beneath)

    After you have downloaded it, you will just require to stick to the quick
    tutorial I have created under!

    After you have completed that, you will be 1 phase nearer to receiving 1000's of Followers!!