Maintenance mode, aka “fix your muckups”
December 10, 2009
or …I already built it…you mean I have to maintain it, too?
Computers and IT infrastructure are a lot like any other complex life forms. They are created, they grow, and they eventually die. This life cycle is universal, and all organisms experience it. Our infrastructures, although not biological, are no exception.
I tend to view my infrastructure as generational. The infrastructure that I have right now is not the same as the one I had 4 years ago, and it’s not the same as the one that I’ll have in 4 years, even though it’s the same company, and it accomplishes very similar tasks. If you were familiar with the previous generation, you would see a definite family resemblance in the network as it stands now.
In between the stressful and joyous times of building and deploying new infrastructures, we’ve got to make continual changes to the infrastructure to ensure its good health. As I mentioned in the link above, we administrators provide the homeostasis that the infrastructure can’t provide for itself (yet).
One of the changes that my infrastructure is going through right now is the growth of scope. I’m being requested to nearly double the number of machines available to produce reports, and contemplate increasing the number of application servers that display those reports for our clients. If only this were as easy as buying more machines.
Essentially, I’ve got to expand the infrastructure under the machines themselves. When we do infrastructure upgrades that change a decent percentage of the existing footprint, we really owe it to ourselves to examine what impact this will have on the rest of the interrelated systems. In this case, I’m going to have to determine what the power requirements are, and if we’ll need to have more electricity at our disposal, I need to make sure our switches have the extra ports, and I’ll need to ensure that we can get new licenses for some of the software that we use.
And then, because I try to run as highly available of an infrastructure as I can, I have to do the same thing at my backup site. I can never buy just one machine, or upgrade just one of the sites. The change has to be universal across the infrastructure.
I don’t currently have a checklist for this sort of thing, because I’ve never done it before, but doing something for the first time provides a great change to make a checklist and document as you go, so that when it happens again, it’s easier.
The documentation also helps because I know that someday, I’m going to have to build a new infrastructure to replace this one.
Posted in General
Email me
content rss
December 10th, 2009 at 3:02 pm
i’ve been trying to work some real extensive backups into our system. That was until 2 days ago when the head of our european IT group thought it was better to cherry pick which bits of user data where backed up. Apparently 5Gb of space is enough for any developer and their virtual environments can be disregarded.
The thing thats disturbed me the most is that he seems to think we can move our 3 (soon to be 4) esx machines to a shared store and then back that up to the local drives where they sit now. I’m a huge believer in offsite backups (would be at home too if i had the money to move that much data elsewhere), has the world changed direction and left me behind?
December 10th, 2009 at 3:31 pm
chewy_fruit_loop – I don’t think the world has left you behind; rather, it’s left your european IT group head behind. Such thinking is blinkered, dangerous and lacking in understanding in the role of backup, and you’ve be justified in pushing back on the request, or at least getting the requirements in writing so that when data can’t be recovered later, you’re not to blame.
December 10th, 2009 at 9:39 pm
…ooh yeh point taken. I’m my ahh f*** them rage I’ve been in I completely for got to get the get of jail free email
The weird thing is, the old office manager who everyone berates as a luddite, is more concerned that the backup tapes are usable for recovery than any other person I have contact with. I really respect that in him.
our central IT guys want us to reduce our infrastructure overall…. so its been consolidated to over 65 virtual machines on 3 servers…plus 9 assorted other boxes.
Think I’ve done quite well this year though, we doubled our chiller capacity, got a 250% increase in ups cover
i’m in the process of acquiring a bunch of WORM tapes to offload some older data to, as we just don’t have the storage we need.
I think that other than hardware replacement (running 4 boxes > 7 years old) the only predictable infrastructure expansion I can make is probably 30% year on year storage, is this the same everywhere, or do people have hidden arts of data zapping I have not imagined yet?
December 11th, 2009 at 12:10 pm
These days we spend almost as much time decommissioning old servers as we do deploying new ones..