October 26, 2012
As I write, here's the current landfall forecast for Hurricane Sandy:
As you can see, it's a decently sized hurricane that's projected to make landfall somewhere between the Research Triangle in North Carolina and Boston, MA. The center of that range is the northeastern corridor that has places like Washington DC, Baltimore, New York City, New Jersey, and an absolute TON of datacenters.
What you should be doing RIGHT NOW if you have anything hosted in this area:
- Verify that your offsite backups are good
- Re-read your disaster recovery plans
- Test your backup generator
- If you're hosted in AWS East, (re)test spinning up instances in other regions
If they aren't, ship some today before the overnight cutoff is reached if you have good ones.
Refresh yourself on what needs to happen if the site goes down, what the recovery procedures are, and make sure to remind anyone else involved in the infrastructure recovery to reread the document, too. Make sure everyone's contact information is up to date in the disaster recovery plans. (Thanks Joe Kern!)
I've had a backup generator fail during a big storm, and it wasn't fun. I used the downtime to my advantage, but I'd rather have not failed in the first place. It would also be a great time to call your diesel supplier and to make sure everything is ok there, so that you can reach out to someone else if need be.
Here's a guide from ServerWatch on migrating to another region. The recent outages reiterate the need to treat cloud infrastructures exactly the same way you treat your physical infrastructure: If you wouldn't just have one datacenter location, don't have just one cloud location.
This isn't doom and gloom stuff - all of these are things that you should be doing anyway, but sometimes they slip until it's too late. Don't let it be too late, do them today.
Also, if this doesn't affect you at all, and you have a macabre streak, you might be interested in taking part in Ben Cotton's Hurricane Sandy Landfall Contest.