Backing up config files

As systems administrators, we’re responsible for maintaining a semblance of consistency across the infrastructure. Obviously, devices across the network will probably not match each other identically, but consistency in this case is more than just identical configurations in multiple cases. It’s maintaining a standard documentation profile for every device, in whatever language that device understands. It’s also maintaining a consistent backup policy for that configuration, and a record of previous configurations.

By default, not every device keeps records of the old configurations. Almost every device has the ability to save the configuration as a local file, though. Sometimes it’s in binary, but usually it’s in text. In either case, a subversion repository would be the perfect storage medium. Checking in new configs with notes relating to the changes made are excellent ways to track the configurations of various devices. This also allows you to browse the history of a device’s configuration which might be useful if you can’t find other sources of documentation (When did we get that Qwest line again? No, the one before this one).

Thanks to the flexibility of svn, testing configurations is a breeze with branches available for testing. Connecting an svn repository with a tftp server would allow for excellent flexibility in remote configuration of devices. Set up correctly, it can also manage host configurations without much effort.

It would also be a good way to store public certificates. Distributing the cert to all the machines that needed it in a web cluster would be much easier that way.

Anyway, I suspect that subversion holds a lot of promise as a systems administration tool. At some point, I’m going to investigate it further, and I’ll post the results on this blog.

DNS Changes through ICANN

In case you haven’t heard the news, and probably everyone has, ICANN has passed, by unanimous vote, to “relax” the rules that limit the top-level domains. “Relax” is maybe an understatement. The phrase might be “completely rewrite the internet”.

To make a long story short, for the low, low price of $50,000 to $100,000, you too can create and own a top level domain (TLD). ANY unused TLD, up to 64 characters. As you might imagine, this will shake things up a little more than when they added .mobile and the other > 3 char names.

We don’t have to worry about it until 2009, but it might not hurt to start taking a look at any DNS scripts that rely pre-conceived notions of how domain names should look.

HOWTO: Order a T1

Many administrators out there have networks whose connection to the internet is broadband based. Whether it be cable, DSL, FiOS, or something else, your sole connectivity relies on this service.

For many small locations, this is sufficient. Lots of small offices don’t need the reliability of dedicated circuits, but others can’t survive without it.

If you’re responsible for a network that should depend on the increased reliability of a T1 (or multiple T1s), it’s your responsibility to figure out how to get that taken care of. That’s where I can give you some pointers.

Over the course of my current position, I’ve ordered, as near as I can tell, around 6 or so T1 circuits (or DS-1 circuits, as they’re sometimes called). Every one is a little bit different, but similar enough that with some pointers, you can feel confident ordering them as well.

A primer on T1s

You’re probably familiar with normal broadband services such as cable and DSL. They sell both business and home accounts, but they’re both the same idea, and they’re both “best effort”. What that means is, you’re not guaranteed any amount of bandwidth at any particular time. The speed quotes you get from the advertising are theoretical maximums, and are dependant on the prevailing traffic of your section of the provider’s network. There are also no Service Level Agreements (SLA’s), which promise you a certain percentage of 9’s (as in 99.999% availability). In terms of cost, broadband typically costs somewhere between $40 and $120 dollars, depending on the level of speed, and the services the provider lets you run on your connection.

Contrast that with a T1. A T1 is a dedicated circuit between you and the telephone company. It runs at 1.544Mb/s both ways, always. Provided there are no malfunctions with the equipment, you are guaranteed 1½ Mb/s constantly. In addition, you get an SLA guaranteeing that your service will be available a certain percentage of the time. The cost of this type of reliability is much higher than broadband, usually totaling between $450 and $800, depending on your location. A T1 is completely unfiltered, allowing you to run any service that you want on your connection.

Step 1: Looking for a carrier

When it comes to buying a T1 circuit, you’ll end up getting the physical circuit from the local telephone company, and possibly the digital signal from another carrier. Neither of those companies may end up being the one you pay for the service. Here’s why.

Reselling T1 service is a big business. Big enough that there are even companies dedicated to finding you providers, and aggregating price quotes from them to help you compare and contrast their services.

The actual phone lines are owned by the Incumbent Local Exchange Carrier (ILEC), and by antimonopoly laws are required to provide access to Competitive Local Exchange Carriers (CLECs). In the case of my New Jersey lines, Verizon is my ILEC, and my service is provided by AT&T (the CLEC). In Ohio, my ILEC is Windstream, and my provider is Qwest.

To make matters more interesting, or difficult, lots of large Tier 1 providers resell their T1 services to smaller companies who buy them in bulk. Since the services bought in bulk are cheaper, the savings can be passed along to the purchasing companies (that’s you).

This knowledge comes as a double-edged sword, however. The service provided by these resellers may be cheaper, but customer service can definitely take a hit. If you buy from a reseller, you may end up with a Tier 1 provider’s line, but unable to contact the Tier 1 provider for support, since you’re really the customer of the reseller. I’ve been in this situation, and it’s very frustrating. I don’t recommend using a reseller unless it’s the only way you can afford the circuit.

Step 2: Comparing the offerings

Assuming you’ve either gotten several quotes yourself or used a quote aggregator, you’re looking at a lot of dissimilar offerings. Here are tips on making sense of them.

First, make sure you’re comparing apples and apples. You’ll be getting quotes for “managed” and “unmanaged” services. The only difference is that with a managed service, the provider (or someone contracted by them) provides and manages the router which is used to connect to the T1. Go with an unmanaged service and you’ll be expected to provide (and troubleshoot) the endpoint equipment.

If you’re familiar with routers, this is probably the best option for you. An older Cisco 2611 router can be purchased used for the price of 2 months of “managed” service. If you’re not familiar, managed might be worth the money.

Make sure you compare contract terms evenly as well. The usual minimum term limit is 2 years, and most places will give you a discount if you sign for three. If you decide to save the additional money, make sure you know under what terms the contract can be prematurely terminated. We once cancelled a contract in the middle of a 3 year term because they decided to raise circuit rates on us.

You’re also going to be seeing people quoting full price, as well as the broken down pricing of “loops” and “port” charges. It sounds complex, but it’s pretty simple.
Since the T1 is a dedicated circuit, it needs a dedicated port. The ILEC, who owns the lines, as well as the machines the lines plug into, charges a per port fee. It’s usually $150-$300, though I suppose in some places, it could be a little more or less.

The loop charge is the cost of running the digital circuit from the telephone office (telco) to the site. This fee varies based on distance from the office. I’ve seen it from next to nothing ($100 on the same floor of a co-location) to $600 (or more). If you’re remote, then this is the part that’s going to cost you.

If you absolutely need the service, but the price is a little steep for you, the option exists to get a “fractional T1”. In my opinion, it’s really not worth it, as the minority of the cost is the bandwidth. You’ll still be paying full price for the port, and the majority of the loop.

Step 3: Ordering the T1

Actually getting the circuit ordered is pretty simple, once you’ve got a provider picked out. In most cases, all you need to do is tell them you want it, and sign the contract. If you have need of a lot of externally facing IPs (probably more than are in a /28 subnet), you’ll probably have to fill out an IP justification form, that the provider will accept and deliver to ARIN.

The provider may also ask you where you want the line terminated. The options are usually leaving the connection in the Main Distribution Facility (MDF) on the ground floor, where all the lines come in, or whether you want the line installed to the Intermediate Distribution Facility (IDF), which is usually the telephone closet on your floor. You can also have them run the line to where your equipment is, in your server room. For lots of people, these three rooms are the same place.

It takes time to get a T1 installed, as well. Depending on the level of cooperation between your provider and the ILEC, I’ve seen as quickly as 5 weeks and as long as 3 months, with the shorter side (around 6 weeks) to be average.

Step 4: Getting the T1 installed

Getting to the point where you can use the T1 takes a couple of steps. Your provider should be in touch with you after a couple of weeks to let you know when your FOC date is. This is the day that the line will be physically installed on premesis. It’s necessary that the ILEC’s worker has access to both the MDF on the ground floor of the building, as well as the place you told them you wanted the line installed.

After the technician comes and physically installs and tests that the line is terminated correctly, you will get notice from your provider that you can schedule your “Test and Turnup” date, and you will be given the contact to arrange this with. The “test and turnup” actually activates the “signal” going to your T1.

Step 5: Test and turnup

At this step, you will need the Customer Premise Equipment (CPE) on-site and ready to plug into the T1. The CPE is really just the router that talks to the internet. If you haven’t received the information already, request your IP details from your provider, and they’ll give you the information on how to configure the router.

Chances are that you’ll have a /30 network (/30 allows two usable IPs. One will be the provider endpoint, the other will be your router), and another network, probably /28 (depending on how many IP addresses you requested).

The router that I use most frequently to connect to my T1s is a simple Cisco 2600 series. In addition to the 2600, you need a T1 CSU/DSU Wan Interface Card (WIC), that just slides in the back of the router. Any Cisco refurb dealer should be able to get you these pretty cheaply. In fact, the T1 WIC will probably cost more than the router. I’d expect a few hundred total, but shop around. Prices fluctuate constantly.

If you feel uncomfortable configuring your router for a T1, you can look online for instructions, but there are several “right” ways to do it, depending on how your provider is configured. Get in touch with your contact at the provider and ask them to talk to an engineer. Every provider I’ve ever dealt with was more than happy to help, and had very knowledgable people who could give you advice.

On the test and turnup day, make sure your router is configured, and that you’ve got a couple of extra regular Cat5 cables. It’s also good to make sure that you can access the console of your router, because if something isn’t acting right, you’ll be able to help debug it on your end.


After reading this, hopefully you’re more familiar with the process of ordering and installing a T1. Getting one installed isn’t nearly as imposing after you’ve done it before, but there’s no reason it has to be hard the first time. It’s just another process that most people have never dealt with, and hopefully now you’re more comfortable and know what to expect.

If you have any questions or comments (or if I’ve made any mistakes or forgotten anything), please reply and let me know. Thanks for your time!