August 21, 2012
When I woke up this morning and started thinking about what I had to do today, I realized that I’d never really written about the process of running cables in the server room. Lets fix that!
So, step one is to plan what you need to do. I’d be all sarcastic, and say “obviously“, but to a lot of people, it’s like, “What? We don’t have enough cables? RUN MOAR CABLES!“. Before we go all ‘Leroy Jenkins’ on this, lets sit and consider the answers to these questions:
- Which servers need cables
- How many cables each server needs
- How long of cables do we need (aka “Where is the Machine?”)
- Do we have enough available switch ports?
- Do we have enough cables of the right length?
Once we know these things, we’ll be in much better shape.
In my case, these additional cable runs are a small piece of the larger goal of replacing our main NetApp filer. Some of our virtual machines had been having issues with network IO contention during our regularly scheduled snapvault jobs. Our workaround had been to throttle the process through software, but the new filer has some extra ports, and “gee-wouldn’t-it-be-nice if we used those to create a dedicated channel for the snapvault process?”.
While I’ve got my cable-running hat on, some of the VMware machines were initially configured with redundant interfaces on their management vswitch, but because we didn’t have enough free ports at that time (remember to check #4 above!), each server got only a single cable. So I might as well take care of that, too.
In the case of our brand new shiny NetApp 3210 filer, there are two controllers, and each controller has 6 interfaces (since we added a 4-port card). Four of the ports are bonded together in an LACP trunk which has multiple virtual interfaces so that it can be connected to several VLANs.
That leaves two interfaces for our nefarious purposes, and since we have two controllers, we’ll need 4 cables to the NetApp.
There are seven vSphere hosts, and I manually checked each one to see how the management vSwitch was configured. I found that three of them had been configured to have redundant connections, but of course, there was no cable plugged in. When I take care of that, at least the annoying error icons will go away. The other four needed me to add another interface to the vSwitch, so I did that through the interface. It worked because there were only seven. If I had 70, I’d have been writing myself some PowerShell.
So now I knew how many cables I needed. We don’t yet have a rack-level map of the serer room, but we do have 21 racks…(this lack of a map will be fixed, promise), so off to the server room I went to figure out the distance.
When measuring distances for cable runs, whether in the server room or in an office space, don’t forget to take into account the vertical length. Racks are 42u tall, so make sure to account for where your equipment is. Always use a longer cable than you need to account for bend radiuses and slack.
Of course, some people prefer to make their own cables to length, rather than buy pre-made patches. I understand why they do this, but I disagree that it’s worth the tradeoff. Having an extra coil or two of network cable on top of the rack isn’t the end of the world, but if your hand-crimped end comes loose some day, it’s going to make for very intermittent issues at an inconvenient time. It doesn’t matter how good you are at crimping cables, you’re not as good as the machine that crimps the cables and applies the molded ends. So just buy the cable to length, if at all possible.
As it turned out, I didn’t have enough cables of sufficient length to make all of the runs today, so I settled for the four most important lines (the connections to the NetApp). Which was actually pretty good, because finding open ports on our switch was…troublesome.
As an aside, right now, we don’t really have a network “core” like you would think of one. We’ve got a big 6509 switch that everything in the server room plugs into, including the uplink to the rest of the university. Although it does work, it’s definitely not optimal. The network layout that I want to go for is this one:
What this gives me is better resiliency, it has the capability of better performance, and I can expand a lot more with this. Plus if a “core” switch dies, the other one is still alive and kicking, and will take over.
But I’m not there yet. Some day. Until then, I’ve got to deal with a current lack of network ports…so I only ran the four. But I didn’t just grab four cables and climb into the ceiling. Oh no, we’ve got to get them ready.
The first thing I did was to determine where on the switch I’d plug them in, and I documented which cables would be going to which ports. Then I took the cables out of the bags, left them in the coil, and used a cable tester to make sure that they were good. The chances of a machine-made cable being bad are low, but not impossible, and I’d prefer to only climb through the ceiling once a day, if possible.
After the cables tested good, I still didn’t unwind them. I labeled both ends of the cables (do you hear me? BOTH ENDS) because this would be the last point where it was obvious which cable ends were matched. As soon as you unwind the cables, you have to start tracing if you want to figure out which end of the cable matches the other. Label them before they’re unwound and you’ll never wonder again.
Once the cables were tested and labeled, then and only then did I unwind them. Since I had to use 30′ cables, I needed a place long enough to stretch, so I went out in the hallway.
It would be very good to take this time to straighten out all of the kinds and loops that are still in the cable at this point. You want it to be as easy to run as possible, and you don’t want to have to deal with all kind of crazy loops at the same time you’re trying to not fall off the ladder.
After getting it reasonably loop and kink free, I looped it into loose coils and trudged to the server room, where I began by evaluating how the existing cable runs to that rack were accomplished. I noticed that there was a small loop of extra cable by the NetApp filer, but that the bulk of the excess was at the switch-end on top of the racks.
I put the loose coil of cable on the ground near the rack with the filer, carried the remote end up on to the ladder with me, and started feeding it across the cable ladder in the direction it was headed.
I feel like I could have done a better job of keeping the bundle together during this time, but I’m honestly not sure what I could have done. I used some twist ties (4 or 5 of them) along the bundle to make sure that the individual wires didn’t get too far out of control, but I still had to deal with a lot of loose wires. I’d be interested in hearing from someone else what they do, and what a better way is, if I’m doing it wrong.
Anyway, I got the bundle up onto the cable ladder going parallel with the existing bundles that were up there. Once a section was in place, I took apart the velcro wrap and put included my new bundle with the old. Here’s a picture of the middle of the process:
The four darker purple lines heading straight down in the middle are the ones I was working with. You can see I’ve got them fed through the brackets (which are overflowing – something else that will be fixed with top-of-rack switches), and I’m getting ready to bundle them into the rest going to the switch which lives in the rack directly underneath the cable waterfall there.
At this point, it’s simply a matter of configuring the switch ports with the right settings and we’re done.
Cable running isn’t necessarily fun, but it doesn’t have to be terrible, and there are things you can do to make it easier. The biggest one is to just plan ahead, though. I’m very much open to what you think of my process, especially if you have any pointers on how I can improve it. Just comment below to tell me. Thanks!