Bizarre issues almost always point to DNS problems

Date February 19, 2009

Duct tape. The Force. DNS.

These are the things that bind our world together. Sure, you can't see the force when you're juggling rocks while standing on your head, just like you don't pay attention to DNS 99% of the time you're browsing the web, but that doesn't mean it doesn't affect everything you do.

Misconfigured DNS has caused more, weirder problems than any other single aspect of networking I've yet encountered. Sure, it causes plain, vanilla connectivity issues when you can't resolve something, but it gets much weirder.

Misconfigured DNS causes mail to break, active directory to stop authenticating (or to even recognize that domains exist), SSH sessions to timeout instead of connect, and an entire host of other problems.

I have even had it cause password issues: the DNS that I was on pointed to a different machine, yet configured identically and with all the same identifiers, and when someone added my account to the machine she was talking to, I couldn't get access. We fought with this for a few hours before I got desperate enough to check into the IP addresses we were connecting to.

This is just a friendly reminder that DNS is everywhere, and if you're having a bizarre network issue, make sure DNS is somewhere early in your troubleshooting checklist.

  • David

    My favorite DNS incident revolves around mail.

    Got called to a customer site where their mail system was suddenly bouncing with authority every piece of inbound mail on the grounds that the sender was blacklisted. I checked for footprints in the log files, as usually this is caused by the client "fixing just one thing", but no, in this case nothing really had changed in six months.

    Eventually (as in several hours later) I noticed they were using the black list as:

    blaklist.com

    (not literally but one of their lists was misspelled). When 'blaklist.com' doesn't exist on the internet, this is a harmless problem, it just means there is a DNS lookup delay on every inbound message. However, I discovered that someone out there in the internet had registered "blaklist.com" and put a DNS wildcard entry on the zone. Now *.blaklist.com did resolve to something, which means sendmail assumed the sender was blacklisted, and it would dump the message.

    Fix the typo and regenerate sendmail.cf, and everything works again.

  • Matt

    @David

    That's great! DNS is so much fun sometimes ;-)

  • Reamer77

    I totally agree that DNS is often overlooked when trying to find a culprit. To mitigate problems, it's a great candidate to be maintained by configuration management (cfengine, puppet, home-rolled rsync) since /etc/resolv.conf is so universal for *NIX machines.

  • billr

    Some great tools for assessing your DNS setup from Cricket Liu (author of 'DNS and BIND' from O'Reilly):

    http://www.infoblox.com/services/

    There's an 'External' tool (run from your browser), an 'Internal' tool (Windows Exec, in free and $ versions), and audit/onsite options.

  • Scott Seltzer

    This is an old post, but I had to mention this.
    We had a client who was complaining that they would occasionally have outgoing mail bounced back to them. The domains involved were random, and the problem was entirely intermittent. The NDR's always complained about the user not existing.
    Long story short, their ISP provided DNS servers (which they were using as forwarders) were overloaded. MX record lookups would timeout, at which point the DNS server would return the A record address instead, causing the mail server to attempt delivery to their web host.

    DNS is almost always high on the list of things I look at when problems seem unexplainable.