Even though it is more and more common for all of us to use a GPS in our car or even hiking, we often forget that when we send an email, DNS is the GPS system that determines where the email goes to be delivered; and, that when we browse a web site (directly or via a search engine), DNS is the GPS system that tells your ISP where to find the physical server(s) involved in serving the site.

Why does this matter?

Well, if you’ve ever used a GPS device that took you to the wrong destination or otherwise could not find the destination from your location, then you have a frame of reference for what I’m about to write.

DNS, like GPS Systems, can be broken or otherwise faulty. Just like a GPS device, it might work great most of the time, but then sporadic at other times.

Before I write much further, some housekeeping:

When a person registers a domain name, part of the domain name registration process is to list one to several DNS servers.

Did you notice the word, “servers” above? DNS is handled by servers; and the DNS hosting provider of those servers might be the company with whom the party registered the domain name, or it could be their web hosting provider, or it could be yet another provider.

While more than one DNS server can be listed, one of the common myths on the Internet is that the first DNS server listed is “the primary,” the second one listed is “the secondary,” and so on giving the extremely false impression that “the primary” DNS sever is always used unless it fails, then the secondary will be used.

Whenever you browse a web site, send an email, behind the scenes there’s a math formula going on as to which of the DNS services listed for a domain name will be used. Then if the one picked (and it might be the secondary or tertiary) fails, after x amount of time, it will try another one (not necessarily in order).

If you have a bad DNS provider, then it doesn’t matter how wonderful your hosting provider is at serving your site or your email provider at sending and receiving email… visitors will not be able to get to the site, email will be delayed or bounce; and if the DNS provider did not secure their servers, then traffic that should be going to your site may be redirected to malicious sites.

Now, how do you know if the DNS servers of a given domain name (the latter part of the @ in an email address) are working and secure?

The following is partial list of sites provide free DNS Reports:

• http://intodns.com/
• http://www.dnssniffer.com/
• http://www.checkdns.net/quickcheckdomainf.aspx
• http://thednsreport.com/
• http://www.dnscolos.com/free-dns-report.html

Common errors you may see on a DNS report are as follows:

• The MX (mail exchange) records (which are used to determine where to send incoming email) do not have a reverse DNS entry.
• The DNS server allows for recursive queries which means the DNS server is insecure.
• One or more DNS servers did not respond; this probably means the server or service is down.
• One or more DNS servers have mismatched entries; this means not all DNS severs have the same information for the domain name.
• One or more DNS servers are lame; typically this means the lame servers know nothing of the domain name in question.
• There’s a mismatch between the parent (domain name registrar) and the name servers as far as what name servers are listed for the domain name at the DNS server level.

Let’s go over the impact of the most common DNS errors:

1. All MX records (incoming email) and mail server records (outgoing and potentially incoming email) need to have a reverse DNS entry; reverse DNS is where you can see that 174.36.196.4 points back to dynamicnet.net just as dynamicnet.net points to 174.36.196.4. A failure to have a reverse DNS entry for any record dealing with email means email rejection. If you don’t want your email treated like spam 100% of the time, make sure there is a reverse DNS entry set up for all mail DNS records.
2. If the DNS servers for a domain name are insecure, then that means traffic can be redirected. If this is your site, that means email that should go to you might be hijacked away to another party; visitors to your web site might unknowingly be hijacked to other destinations.
3. If a DNS server is down, and that is the server picked for queries… then you may see email delays, visitors who give up waiting (it does take time for the fail over to another working DNS server to take place), etc.
4. Lame servers can be worse than down DNS servers…. answering they don’t know anything about the domain.
5. Mismatch between DNS servers can be extremely common as this does happen when a (hopefully authorized) party makes changes to the DNS entries for a domain name; but should not exist past for longer than 60 to 120 minutes. If only one name server has the most up to date information, then when the other name servers are queried, over time you see extremely sporadic results.

There’s a lot more about DNS that what I’ve written above, including, but not limited to discussing local vs. public DNS. The main take away’s I hope you leave with after reading this article is that you know what is DNS on the layperson level; and that you understand that whenever a domain name is involved (email, web site, mobile, etc.), you also realize there is a DNS hosting provider hopefully taking care of the DNS services for the domain name.

Please feel free to comment below if you have questions about DNS or want to share your own experiences in trouble shooting DNS issues.

Thank you.

One of the projects we consistently use and recommend is Ryan’s Advanced Policy Firewall by R-fx Networks known as APF

While we do customize the implementation of APF as well as BFD (making some core changes to allow us to integrate APF into our other managed security offerings), one of the issues we run into from time to time with APF is that if local DNS resolution is not working when the server is rebooted, a server will hang at starting APF.

Most of the time this issue can be resolved by making sure local DNS resolution is perfect on reboot.

Sometimes an easy fix is editing /etc/resolv.conf to make use of the data center’s recommended name servers; other times it might be using the free DNS service of OpenDNS or Google.

What about the co-location customer or customer of a small mom and pop data center who cannot assist with how the server integrates with the data center network on a local DNS basis

Well, then you are stuck with a decision of whether to use APF or not, or reboot the server without APF; then someone has to remember to start up APF once local DNS resolution is working.

The latter part is not a good option for security, and the former doesn’t fit for customers who want an integrated picture that includes APF and BFD.

I did ask Ryan about the issue about a year ago (this problem doesn’t crop up often), and Ryan’s reply is accurate and makes sense:

Until now, we’ve worked around the issue using standard techniques, one of which is mentioned above (change what is in /etc/resolv.conf), until recently where one of our long standing customers got a good deal on a MAC-based server; and put it in a data center where everything is “you are on your own other than ping, power, and pipe (which is more typical than not).

While it would be extremely easy and convenient to put the burden of this problem on the customer and the data center they picked, I was determined to not to go through status quo motions.

The crux of the matter was local DNS resolution. How could I test it where the test itself would not hang or otherwise take long to run?

Creating the test is easy, comment out all entries of /etc/resolv.conf

Yet, the common tools recommended over the years such as host and nslookup to check for DNS issues (local and external) do not appear to have a way to control time out other than how often it will retry. Comment out the entries in /etc/resolv.conf and then run host or nslookup against a valid public domain name, then wait… and wait… and wait… and wait… Hmm… isn’t that what is causing the APF problem?

Well all this thought about “digging into local DNS resolution” brought me to the dig command. dig is a part of the bind-utils package available for CentOS through yum installation (yum install bind-utils -y).

Looking at the dig man pages, I saw you could pass a time out and number of tries and retries as part of the command structure. Would this work quickly enough to test local DNS resolution?

Using dig +time=1 +tries=1 +retry=0 yahoo.com, I was able to test response times on various servers when local DNS was down as well as up. The response time appeared to be acceptable within less than or equal to two seconds.

I wrote a test script to test my theory and create a framework for what might be used to restart APF if it was down because it was never set up to start on reboot (to avoid hanging on reboot).

Running through the test proved the concept, but now what… think, think, think. While I could write a wrapper and put it in /etc/rc.local (which is part of the boot up sequence), I would prefer to have the server running a little bit longer, even if by a few minutes; which also meant avoiding using a sleep command.

Then I remembered that we recently created a System Integrity Monitor — S.I.M. — add on module just a few weeks ago to test for IP addresses which became unbound from the network interface. I wrote about that adventure in the elusive hunt to protect against IP blackouts.

Now that I had the code to check if local DNS is working, incorporating the code as part of another S.I.M. add on module was easy.

While the real test will come from our client with the MAC doing a controlled reboot of his server (we have APF turned off for reboot), testing the add on module is as simple as doing the following:

• Everything running fine — APF up, local DNS working –> run sim -s
• service apf stop to shut down APF –> run sim -s
• service apf stop to shut down APF and edit /etc/resolv.conf to comment all name servers –> run sim -s
• service apf stop to shut down APF and edit /etc/resolv.conf to uncomment all name servers previously commented out –> run sim -s

When you are happy everything is working, and want to have APF come up after reboot (not during when it might hang), then run “chkconfig –levels 0123456 apf off” to turn off apf from starting on reboot.

Since Ryan MacDonald is kind enough to share S.I.M. with the world, I thought it would be nice to share dni_apf.mod with other R-fx Network fans.

Please contact us if you have any questions.