Puzzles -vs- Mysteries

Date November 24, 2009

I am a huge fan of Malcolm Gladwell. The first book I read of his was Blink, which talked about the power of snap decisions when made by experienced people. I then read Outliers and (my favorite), The Tipping Point.

I really admire Gladwell’s method of writing, the way he ties several utterly dissimilar threads into a common relationship, as well as the unbelievable depth of his research. I’m often in awe of the dissimilarity between the examples he finds that all rely on the same underlying mechanism. One of the more amazing examples in my mind was his comparison of the popularity growth of hushpuppy shoes to the STD rate in Baltimore. As it turns out, both were stable populations that experienced a “tipping point” event that caused epidemic growth. You can’t make these things up.

When What the Dog Saw came out, I was hesitant to pick it up, just because it was a collection of previously published articles from his column in the New Yorker magazine. I did it anyway, and I’ve been very happy with my purchase, even though all of the works are available on the New Yorker website.

One of the articles contained in the book was called Open Secrets, and the largest part of the article discusses Gregory Treverton’s thought that puzzles and mysteries are distinct from one another.

To briefly recap, puzzles are situations where the issue is utterly solvable, given enough information. There is a well defined solution or answer that, if we have all the pieces, becomes obvious. The example of a puzzle that Gregory uses is the specific question, “What is Osama Bin Laden planning”? While this isn't necessarily the best example, in my opinion, it's reasonable to assume that if we had direct interaction with Bin Laden, we could determine this. An example of a mystery is a situation with a much less clear-cut resolution. Again, one of Gregory’s examples is, “What will Mexico’s inflation rate be this year”? No amount of existing information will give us that answer. We can make better guesses by gathering more information on the variables involved, but the answer is unknowable until it comes about. The takeaway from Gregory’s article is that the old method of learning secrets to solve puzzles is going away, and we’ve now got to concentrate on the dissimilar skill of resolving mysteries.

I’ve wrestled with this distinction in my head for a week or so, ever since I first read it in Gladwell’s article. I’ve tried to find resolution between my basic belief that everything is knowable with enough information and my tacit knowledge that some things are too complex to be conclusively determined, particularly when humans are present in the equation.

So far, this conversation probably appears out of place on a systems administration blog, but each of us confronts puzzles and mysteries every day. Instead of asking, “What is Osama Bin Laden planning”, try asking “why did the mail server drives fill up without issuing a warning”? And instead of, “What will Mexico’s inflation rate be this year”, does “What will the bandwidth requirements be for the next three years”.

Treverton’s application of this concept is confined, for the most part, to national security. The implication was that our three letter agencies are going to have to learn to deal more with mysteries and less with puzzles, because the game is changing. Gladwell used the distinction to examine Enron’s policy of openly publishing its financial data and hiding their shady accounting practices through obfuscation rather than deceit, with the implication that mysteries require us to work harder determine the truth, and that the blame falls upon us as well, if we are unable to solve the mystery.

There are many situations that we deal with as puzzles that would probably be better dealt with as mysteries. IT security certainly comes to mind, as do many capacity planning situations, and pretty much anything that involves user motivation. I think that too often, we over-simply problems in order to make them more manageable. This allows us to take shortcuts, but could it be that we are doing ourselves a disservice?

Mysteries can be unresolved, and they can also be irresolvable. Even at our most enlightened, we have to be prepared for the idea that we will not learn the underlying motivations for everything, but that is no excuse to not try. To give in to defeatism is the wrong attitude. We are computer scientists, and we follow the scientific method and the scientific spirit of learning.

I suspect that I’m preaching to the choir here, but mysteries should be an intrinsic motivation for your work. If you didn’t enjoy solving mysteries and figuring out puzzles, you’re in the wrong gig.

  • http://www.funnelfiasco.com Ben C

    That distinction makes perfect sense, now that it has been pointed out. Sounds like I've got some books to add to my queue (although I vaguely recall reading all or part of The Tipping Point in college). Regardless, I'll be sure to not wear hushpuppy shoes.

  • http://saintaardvarkthecarpeted.com/blog Saint Aardvark the Carpeted

    If you didn't enjoy solving mysteries and figuring out puzzles, you're in the wrong gig.

    Testify!

  • AJ L.

    I enjoy solving puzzles...I don't necessarily like it when others are hounding me to get the puzzles resolved in an amount of time that they set with no idea as to how long it should really take though.

  • http://adminmas.blogspot.com/ MAS

    Mysteries .. You have cleared the view.

    Thank you Matt.

  • http://www.standalone-sysadmin.com Matt Simmons

    @Ben
    I'm not sure that was the message, but it's a good safety measure just in case!

    @AJ
    Ugh, I know that feeling. I have been planning to write a column for Simple Talk about unreasonable expectations.

    @MAS
    Thanks for the kind words, and for reading!

  • Bill B

    If you've enjoyed Gladwell's books, you should definitely pick up Freakonomics. I haven't read Superfreakonomics yet, but I think you'll enjoy their first book.