Don't Panic

Date January 27, 2009

Many thanks to Bob Plankers for this blog entry!

What do you do when you first find out that a big problem is happening? Perhaps your data center has lost power, or your intrusion detection system has told you that a web server has been breached by a hacker.

For most people incident handling is one of the most difficult skills to master. You don't get much practice, and most of the time when you do get practice you're actually in the middle of a crisis. You may not have as much information as you need. You might have customers and managers contacting you, yelling at you, wanting to know what's going on, when things will be back up, who is fixing it.

The best advice I've ever been given is simple: don't panic, calm down.

Laurence Gonzales at the National Geographic Adventure Blog describes panic quite well:

"Panic was really useful to us once. We have inherited the structure and function of our nervous system from ancestors who lived in a very different environment, where simple, automatic actions were required for survival. A form of panic -- running away or fighting without thinking, for example -- was apt to keep them alive more often than not."

"But when that quick action also requires logical thinking -- when, for example, you're breathing underwater using a complex apparatus -- it can sometimes incapacitate us. Remember, the higher the emotion or stress, the lower the ability to think in a step-by-step fashion. There are three important steps to take to suppress panic: Breathe, organize, act."

That's excellent advice: breathe, organize, act. Panic causes others to get alarmed, which in turn degrades their ability to think logically with you. You end up wasting time fighting the chaos you've created, rather than calming finding facts and evaluating options. Sometimes you might find, as in my earlier example of an intrusion detection system alerting folks to a breach, that all the panic was for nothing, as it was a false alarm. If you're calm, and the people around you are calm, you can rapidly move on to assessing the facts of the situation, assigning different roles to people, and fixing the problem.

Breathe, organize, act. Great advice for anybody, especially when there's a problem to be solved.

  • sysadmn

    I like the advice given to a new resident (MD in training):

    "In a Code Blue, the first thing you do is take your own pulse".

    That's from The House of God, the novel that became the TV series St. Elsewhere.