Backing up config files

Date June 30, 2008

As systems administrators, we're responsible for maintaining a semblance of consistency across the infrastructure. Obviously, devices across the network will probably not match each other identically, but consistency in this case is more than just identical configurations in multiple cases. It's maintaining a standard documentation profile for every device, in whatever language that device understands. It's also maintaining a consistent backup policy for that configuration, and a record of previous configurations.

By default, not every device keeps records of the old configurations. Almost every device has the ability to save the configuration as a local file, though. Sometimes it's in binary, but usually it's in text. In either case, a subversion repository would be the perfect storage medium. Checking in new configs with notes relating to the changes made are excellent ways to track the configurations of various devices. This also allows you to browse the history of a device's configuration which might be useful if you can't find other sources of documentation (When did we get that Qwest line again? No, the one before this one).

Thanks to the flexibility of svn, testing configurations is a breeze with branches available for testing. Connecting an svn repository with a tftp server would allow for excellent flexibility in remote configuration of devices. Set up correctly, it can also manage host configurations without much effort.

It would also be a good way to store public certificates. Distributing the cert to all the machines that needed it in a web cluster would be much easier that way.

Anyway, I suspect that subversion holds a lot of promise as a systems administration tool. At some point, I'm going to investigate it further, and I'll post the results on this blog.

  • Jon

    SVN has a lot of potential as a systems management tool, but there are other more specialised options as well.

    I can't speak from personal experience, since I'm a developer with an interest in systems administration, rather then a professional sysadmin, but I know that the folks here are using Puppet to manage large clusters of more or less identical machines.

    It even allows them to plug in some unconfigured hardware, and have it boot from the network, configure itself appropriately (through some Puppet configuration files), and add itself to the cluster.

  • Matt

    Jon,

    I've heard a lot of good things about puppet. I'm looking forward to spending some time researching it, just as soon as my current system rollout is complete. From everything I've heard, it's a solution a lot of people have been looking for.

    Thanks for the comment!

  • James

    The tool you want for this is RANCID. The only downside would be if you are dead set on using subversion. It uses CVS by default, though I think it can use RCS.

    I've got it running using CVS, paired up with ViewVC so that I've got a web interface to look at the configurations, and it's running on Ubuntu Server 8.04.

  • Michael Janke

    Matt -

    We've been doing what you've suggested with pretty good success. Most of our UNIX config, and all J2EE app server config is SVN'd.

    We've even taken the J2EE app servers to the extreme of having the developers check an entire J2EE application environment, including the JBoss application, all configuration, startup scripts, binaries, properties files, war, ear and jar files into an SVN repository dedicated to application deployment. (i.e., if JBoss is installed in /apps/jboss, that entire directory structure is checked into SVN by the development group that owns the application.

    To deploy a new version of the application, or to deploy the current version of the application on new servers, we (the sysadmins) simply check out the entire application and drop it on the server.

    It works pretty good. We always know exactly what code is on an app server, and we have a complete record of every bit of code and config that was every installed on every app server.

    It's possible to take that concept even further. See: Self Deploying Servers"

  • Matt

    Michael,

    Wow, that's a great sounding setup you've got there! The extent to which I've implemented version control (using our pre-existing cvs server) is DNS config files. You've got the system down pat!

    From the sound of your blog, I bet you're salivating at the thought of the solid state drives coming down in price. Heck, I love the fact that I can go to a computer store and for $14, have 4GB of solid state storage, no DVD required.

    It's going to get exciting pretty soon!

  • Michael Janke

    Matt - The work we did on on self-deploying servers was done a couple years ago. You are right, if we did it today, we could have extremely lightweight, flash based servers to use as 'appliances' at our remote sites.

    Thinking......thinking......thinking......Ding!

    Eepc's are cheap, boot from flash, and have a built in UPS! (the battery).

    Dang - It might be fun to resurrect that project. ;)