The more customer sites I saw, the more I realised that the mainframe teams always had it right – uptime is not about continuous availability, but continuous negotiated availability. Midrange system administrators in particular have too often become hung up on perfect uptime records, when in actual fact a bit of downtime now and again can sometimes be good – for performance, for maintenance and stability, etc.

Here’s an example – a customer once (a major financial institution in Australia) had some business improvement consultants in, and they surveyed the IT people and asked them what their system availability percentages were like. The general consensus was around the 90% mark. On the other hand, the end users, when asked, presented a strikingly different average – around 60% or perhaps even less. The main reason it was determined for the difference was that the IT people were measuring uptime, whereas the users were measuring responsiveness.

Uptime by itself isn’t a good enough system metric. Negotiated uptime, on the other hand, is much better.

I do think you introduce one fail as part of the solution, and that’s that lightweight SAN you talk about. The SAN becomes a single point of failure (and they do/will fail!), and even SANs need upgrades. Because more things rely on them, both these things cause hassle.

I prefer local storage, aka shared nothing. And I make servers not just redundant but expendable. That is, if one fails completely, there’ll be at least one (if not more) copies of both data and processing power elsewhere. Then you can just take out anything without a worry.

Its all about service availability, I constantly reboot servers during production hours but thanks to good redundancy, nobody will ever notice. In fact, the services have been more reliable. The actual method employed here depends on the service but most use load-balancers.

Other than security there are many more reasons to reboot. If you never reboot a server, how will you know that it will boot correctly? I’ve seen servers develop faults on parts of the disk that were only used during boot. I’ve often encountered sysadmins who install some service, start it manually and forget to add it to the startup scripts.. these are not the kind of things you want to be dealing with at 3AM when a power outage causes that server to reboot..
Another advantage of redundant servers is that you need some way of keeping your configuration in-sync, which often involves some kind of configuration management. This will eventually result in faster repair times when something goes wrong.

Proper load-balancers are required to make many services truly redundant. But I will never let anything with a Barracuda badge inside the datacenter. If you are going to load-balance many critical services, do you really want to pull all traffic through some cheap box? I’d go for one with a big red light on the front. It may be more expensive, but its worth it.

the only real thing i need to do is keep everything up for about 12 hours a day, the rest of the time its unlikely people will notice an outage.

but we’re now starting to serve data to sites that are following the sunlight but its not a big deal for the most part

I have aways been sceptical about people who brag about system uptime. That a box itself is online for years never meant the services that the box provided where available to users for the same amount of time.

System uptime is just what what the word implies, system uptime. I am much more interested in service availability, though it is nice to have triple digits when i typ my ‘uptime’ command.

One of the things that needs to be remembered in this profesional and polished environment we live in, uptime be it measured as pure uptime or available time, while important, can be completely undone by bad time to repair.

Sure it’s impressive that someone’s machine has 456 days of uptime. We already know that this means there’s a lack of patches and “maintenence” done to the system. So what happens when the sprinklers go off? What? You’re gonna quit? No you’re not, you’re like the rest of us and you’re going to spend the next sleepless nights trying to reconstruct what was running instead of rebuilding the machine using current patch levels and then copying over the customer’s applications/webpages/other from the backups, looking like a genius and going home at a reasonable hour.

I used to be a projectionist and got taught that “The difference between amateurs and professionals is that professionals make their mistakes quietly.” I also learned that speed of recovery is the measure of a professional.