Category Archives: Networking

Posts relating to IP networking

Great Open Positions at Northeastern CCIS

I’ve landed in Los Angeles, and I’m getting settled in temporary housing until I find my own place, but it’s been a really busy couple of weeks, and I just realized that I didn’t get a chance to post about the open positions that my (now former) team has.

First, more obviously, there’s my old position, that of the Networking & Virtualization Administrator. The position is officially posted on Northeastern’s Careers page, but I can tell you that you’d be responsible for a medium-sized relatively flat network infrastructure. There are a few dozen VLANs, all statically routed from the core switches, and around a thousand lit switchports. The hardware is mostly Cisco Catalyst, with the core being Cisco Nexus 5548s, although there are some virtual PFsense boxes running around too.  You would be working with the (pretty friendly and competent) central ITS network admin to coordinate staff and faculty moves around the infrastructure, and with the university’s security officer (who is also surprisingly friendly, given his line of work) whenever something weird pops up.

The role is also responsible for the VMware cluster, which currently consists of around 15 ESXi nodes and two vCenter instances (one for “production” use which has the vSphere Essentials Plus license) and the educational cluster, built out using VMware Academic licenses for classroom and academic use. They’re backed by NetApp and Nimble storage, and it’s this part of the job responsibilities that gives you a little more creativity to solve problems, since professors usually want interesting things. I’ve built some useful stuff in PowerShell, but there’s no reason you have to use that long-term, if you want to solve the problems yourself.

Anyway, I really enjoyed my time in this position, and to be honest, I really miss the other staff members and students there.

In addition, the CCIS staff is growing. We got a new dean a little over a year ago, and one of the things she wants to do is to offer management of researchers’ clusters in a more active manner, so we are looking for another Linux sysadmin (pretty much all of the researchers do work on Linux).

This position will involve a lot working with our current Linux admin to bring over the technology he has built to deal with our “managed” machines to help with our “unmanaged” or “soon to be managed” researcher-owned machines. Basically, there’s nothing like this right now, so you would be inventing the role as you go. Exciting! Challenging! Rewarding!

Anyway, please, if you’re looking for a position in Boston somewhere, take a look at Northeastern. It’s easy to get to, there’s free tuition for you, your spouse, and your children, and I feel like the staff that I worked with there are my family, and I miss them :-)

If you have any questions, please drop me an email and I’ll be happy to help. Thanks!

Annoying pfSense Issue with 2.15 -> 2.2 Upgrade

I run several pfSense boxes throughout my network. Although the platform doesn’t have an API, and it can be a pain to configure manually in certain cases, it’s generally very reliable once running, and because it’s essentially a skinned BSD, it’s very easy on resources. There’s also a really nice self-update feature that I use to get things to the newest release when they’re available.

It’s that last feature that bit me in my butt Sunday night. After doing the upgrade at midnight or so, I went to bed after everything seemed to work alright, but then this morning, I started getting reports that people couldn’t log into the captive portal that we use for our “guest” wired connections.

I thought, “That’s strange…everything seemed to work after the upgrade, but I’ll check it out”, and sure enough, as far as I could tell, all of the networks were working fine on that machine, but there was no one logged into the captive portal.

Taking a look at the logs, I found this error:

logportalauth[42471]: Zone: cpzone – Error during table cpzone creation.
Error message: file is encrypted or is not a database

Well, hrm. “Error during table cpzone creation” is strange, but “file is encrypted or is not a database” is even weirder. Doing a quick google search, I came across this thread on the pfSense forums where someone else (maybe the only other person?) has encountered the same problem I have.

As it turns out, prior to version 2.2, pfSense was still using sqlite2, but now, it’s on sqlite3, and the database formats are incompatible. A mention of that in the upgrade notes would have been, you know, swell.

The thread on the forums suggests to shut off the captive portal service, remove the .db files, and then restart the service. I tried that, and it didn’t work for me, so what I did after that was to shut down the captive portal (to release any file locks), remove the db files, and then from the text-mode administrative menu, force an re-installation of pfSense itself.

Although I haven’t actually tested the captive portal yet (I’m at home doing this remotely, because #YOLO), a new database file has been created (/var/db/captiveportalcpzone.db) and inspecting it seems to show sqlite3 working:

[2.2-RELEASE][root@host]/var/db: sqlite3 captiveportalcpzone.db
SQLite version 2014-11-18 20:57:56
Enter ".help" for usage hints.
sqlite> .databases
seq  name             file
---  ---------------  ----------------------------------------------------------
0    main             /var/db/captiveportalcpzone.db
sqlite> .quit

This is as opposed to some of the older database files created prior to upgrade:

[2.2-RELEASE][root@host]/var/db/backup: sqlite3 captiveportalinterface.db
SQLite version 2014-11-18 20:57:56
Enter ".help" for usage hints.
sqlite> .databases
Error: file is encrypted or is not a database

What I don’t understand is that the normal way to convert from sqlite2 to sqlite3 is to dump and restore, but it doesn’t look like this process did that at all. It would be incredibly easy to do a database dump/restore during an upgrade, ESPECIALLY when revving major database versions like this.

Anyway, this kind of experience is very unusual for me with pfSense. Normally it’s “set it and forget it”. Hopefully this will work and I can get back to complaining about a lack of API.

VLAN Translation on a Nexus 5548 – :Sad Trombone:

I’ve got a problem. Our school is expanding, and we’re constantly hiring people. We’re hiring so many people that they won’t actually fit in the building we’re in. Because of that, we’re having to expand outside of the building we’ve been in for years. Part of that expansion is extending my networks across campus (and in some cases, farther).

The network that I run is really old. Like, it actually predates the network at the central university. I’ve got around 50 VLANs, and now that we’re growing outside of this physical environment, I’ve got to extend those layer 2 broadcast domains to the other buildings. I have a good relationship with the central network folks, and although most of my VLAN IDs collide with theirs, they assigned us some IDs that we can use on their infrastructure. Now, I just have to translate my VLAN IDs to their VLAN IDs.

My network core is a pair of Cisco Nexus 5548s. When I was planning this migration, I didn’t worry at all, because the documentation clearly declared that the switchport vlan mapping command was supported. The only weird thing was, when I went to set up the VLAN translation, the command wasn’t found. It was in the docs, but not in the CLI. Weird, right?

So I did what you do when you pay ungodly amounts of money for Cisco support: I opened a ticket with the TAC.

I had been operating under the assumption that my device would be able to perform VLAN ID mapping on an interface, but I can’t figure out how to do it.

Is it possible to map VLAN IDs across a link? I have a trunk to my provider across which I need to send several vlans, but my IDs collide with those in use there. I was hoping to use the equivalent of “switchport vlan mapping”, but it doesn’t appear to be in my release.

Can you please advise me?



I got back what may be the best response from tech support ever. Emphasis my own:

Hi Matt,

My name is XXXXXXX and I will be assisting you with the Service Request 633401489. I am sending this e-mail as an initial point of contact and so that you can contact me if you need to.

Problem Description
As I have understood it, “switchport vlan mapping” command does not exist in 5548

If you look at the release notes of Nexus5500

VLAN Translation
Allows for the merging of separate Layer 2 domains that might reside in a two data centers that are connected through some form of Data Center Interconnect (DCI).

So I can understand why you were under the impression that this platform supports this feature however I must state that the document is incorrect here.

I have verified with the Technical Marketing Engineers and it has been confirmed that there are no plans to support vlan mapping / translation on Nexus5500 platforms however as of today; Nexus5672, Nexus 6000 and Nexus7000 do support this feature in 7.x release.

Please let me know if there is anything else I may assist you with .

So that was, you know, less than helpful. And I still need to get those VLANs over there. How are we going to do this?

For now, I’m doing it the old fashioned way. Crossover cables.

Normally, when you move VLAN traffic around, you use a dot1q trunk. Each layer 2 frame gets a header when it leaves a switch that tells the remote device (usually a switch) what VLAN the packet belongs to. So, VLAN ID 10 gets a header that says “this frame goes to VLAN ID 10”, which allows traffic from VLAN 10 and VLAN 20 to be sent over the same physical link and still be kept separate.

Since the VLAN ID is encoded in the frame, it’ll cause problems if the VLAN ID I’m using means something else to the other network. But, since the only thing the other end cares about is the VLAN ID, if I can send my traffic over to the other network on the proper VLAN ID, then they’re happy. To do that, I need to bridge the networks. The easiest way I know how to do that is to take an access port on VLAN A, and an access port on VLAN B, and plug a single cable into both of them (after disabling spanning tree, of course). Yes, this sounds insane. Yes, it might actually be insane. But this is how I did it, and it worked the first time.

The bad part is that I’m currently burning two physical ports for every VLAN I need to translate, and this isn’t tenable over the long-run. Fortunately, the Juniper switches on the remote side of the network link support translation, so I believe that we should be able to do it the “right way”. The sooner the better, because I feel dirty.