Hurricane Sandy Prep Steps

(A reader has taken the time to translate this post into Russian)

As I write, here’s the current landfall forecast for Hurricane Sandy:

As you can see, it’s a decently sized hurricane that’s projected to make landfall somewhere between the Research Triangle in North Carolina and Boston, MA. The center of that range is the northeastern corridor that has places like Washington DC, Baltimore, New York City, New Jersey, and an absolute TON of datacenters.

What you should be doing RIGHT NOW if you have anything hosted in this area:

    1. Verify that your offsite backups are good

If they aren’t, ship some today before the overnight cutoff is reached if you have good ones.

    1. Re-read your disaster recovery plans

Refresh yourself on what needs to happen if the site goes down, what the recovery procedures are, and make sure to remind anyone else involved in the infrastructure recovery to reread the document, too. Make sure everyone’s contact information is up to date in the disaster recovery plans. (Thanks Joe Kern!)

    1. Test your backup generator

I’ve had a backup generator fail during a big storm, and it wasn’t fun. I used the downtime to my advantage, but I’d rather have not failed in the first place. It would also be a great time to call your diesel supplier and to make sure everything is ok there, so that you can reach out to someone else if need be.

    1. If you’re hosted in AWS East, (re)test spinning up instances in other regions

Here’s a guide from ServerWatch on migrating to another region. The recent outages reiterate the need to treat cloud infrastructures exactly the same way you treat your physical infrastructure: If you wouldn’t just have one datacenter location, don’t have just one cloud location.

This isn’t doom and gloom stuff – all of these are things that you should be doing anyway, but sometimes they slip until it’s too late. Don’t let it be too late, do them today.

Also, if this doesn’t affect you at all, and you have a macabre streak, you might be interested in taking part in Ben Cotton’s Hurricane Sandy Landfall Contest.

  • I would add to the list: Make sure you have a paper copy of up-to-date contact information for your team members. Cell phone numbers, pagers, and addresses. Being able to contact your team or management can make things much easier. Especially when the lights go out.

  • Great idea, Joe. Added!

  • Eric

    I would also add vendor contacts to the list as well.

    And lastly money. How much money are you authorized to spend in personnel and equipment to keep your datacenter/servers going.

  • David Tower

    Along with the printed contact list for your company add your primary vendors and any direct contacts you may have. I also started adding things like circuit IDs for leased lines just in case you need to call and check on something.

    I would also put together a list of what order to power your servers on in just in case you go dark AND run out of fuel. Booting everything in the right dependency order can save a lot of headache and troubleshooting after the fact.

    Switching your boss (and his/her bosses) over to cached profile mode in Outlook (with Exchange) so they have all their emails “safely” on the laptop just in case power goes out to your Exchange server. This is a security blanket for Layer 8 more than anything though.

  • I couldn’t agree more about having a paper copy of your service providers & vendors phone #’s along with circuit ID’s.

    Coming from the service providers side I can name a # of times I’ve had panicked customers advising they have services down but unable to provide circuits (or even addresses at times). Not that the providers can’t find them (or at least I hope that’s not the case) but there’s a good chance that if you’re down during a storm, many others likely are as well and the more info you can provide up front the quicker the situation can be looked into.

    If you have a multiple site WAN network with your service provider ask them for a WAN Topology map of your services with them. If they don’t have one for you…tell them you want one created. It’s something that in my opinion a good service provider should be able to provide you.