| This 81 message thread spans 3 pages: 81 (  2 3 ) > > || |
|DNS Recursion - Open DNS Servers|
The new open relay problem. Are you addressing this?
It appears that DNS Report, one of my all time favs for performing QC on websites, has recently changed the results of its DNS Testing Service
Apparently something that has been going on for years has now come to a head and it has to do with DNS Recursion. I'm not a DNS specialist so please bear with my terminology and correct me if I'm wrong, that's the only way I'm going to learn! ;)
The New Error Most Will See
|ERROR: One or more of your name servers reports that it is an open DNS server. This usually means that anyone in the world can query it (it is possible that the DNS server advertises that it does recursive lookups when it does not, but that shouldn't happen). This can cause an excessive load on your DNS server. Also, it is strongly discouraged to have a DNS server be both authoritative for your domain and be recursive (even if it is not open), due to the potential for cache poisoning (with no recursion, there is no cache, and it is impossible to poison it). Also, the bad guys could use your DNS server as part of an attack, by forging their IP address. Problem record(s) are: |
Server xx.xx.xx.xx reports that it will do recursive lookups.
It appears the change in the DNS Tool report has caused some heated discussion from a few who use the tool regularly. Many do not like to see that FAIL message when they've had an Open DNS Server for as long as they can remember.
To make a long story short, having a DNS server that allows recursion for the Internet is like running an open SMTP relay.
|The Good Guys have found over half a million. The Bad Guys will find yours. Bad guys will use spoofed UDP packets to fire-and-forge large DNS requests, and the recursing server will send the fragmented replies to the victim (the forged source of the UDP). There are rumblings by very smart people to "do something about this". |
|A few days ago, it was pointed out that there was a new attack using open DNS servers as part of a DDoS, using amplification (where sending the open DNS servers would send packets 50 times as large as the ones that were originally sent), making this a very serious issue. |
Are you aware of and/or are you doing anything about this? I'm in the process now of discussing all of this with my server administrators and want to make the changes to eliminate that failure on the DNS Report. Anytime I see red on that report, the hair on my neck rises. Many of the issues we see here at WebmasterWorld can be traced back to DNS Issues so it is important that you keep a regular eye on what your DNS is up to! ;)
Brett, congratulations, you passed for this particular test. Over 75% of the sites out there do not, including my own which I am in the process of correcting.
Hmmm, no replies yet. I wonder, am I speaking a foreign language when it comes to DNS? For those of you reading this topic, do you know what DNS is?
DNS = Domain Name System
DNS Problems usually are responsible for a majority of the issues we as marketers and technology providers will face.
I knew absolutely nothing about DNS up until a few years ago when I got intimately involved with the hosting side of things. Wow, did that open my eyes and expand my horizons. Just acquiring a basic understanding of DNS and what purpose it serves is enough to help you be proactive instead of reactive when it comes to DNS issues.
Let's take this whole DNS Recursion issue. For many, this may all be Greek. For some, they are in the process of making the changes required to address this very severe danger.
According to government statistics, over 75% of DNS servers out there are set up for DNS Recursion.
|Note: Reading the below document from the government will give you a basic understanding of what this is all about. It also provides suggestions on how to correct issues on both Windows and Unix platforms. There are plenty of references to review that discuss this very important issue. |
What does this mean?
It means that your DNS Server can be used to perform what is referred to as a DDDoS Attack. You've probably heard of the DDoS Attacks that have been on the rise over the past year or so. Well, this is a new kind of vulnerability. Actually, it isn't new, it is just now coming to light after a recent attack where DDDoS was used.
With DDoS, a Distributed Denial of Service attack, multiple computers are used.
With DDDoS, a DNS Distributed Denial of Service, DNS packets are used.
The end result? When the attack starts, you'll know. Your websites will come to a grinding halt. Your email will be non-functional. You will be basically DIW as they say in the Navy, or Dead In The Water.
And, based on my experience with DNS issues, this is far more severe than DDoS, at least based on what I've read over the past 24 hours concerning the issue.
Are there any DNS Gurus out there who can add to this possible threat?
OK, for the non-techies out there, here is what might happen when you let everyone use your DNS server as a resolver.
For example, I'm a spammer. I create a hostname for spam, something.myspam.tld.
After that, I query your DNS server to resolve the host something.myspam.tld. It resolves that host and stores the info in its cache. All further requests for that host do not result in propagated lookups but are answered by the server from its cache, until the exipration for the zone occurs.
After that, I set your DNS server as the authoritative server for my zone with the domain registrar.
After that, I start sending spam.
Now, you are screwed because it looks as if your DNS server is being used for spam.
There is nothing new about this. It's been like that for years.
|There is nothing new about this. It's been like that for years. |
True. But now that the DDDoS vulnerability has been targeted recently, how long will it take before it makes into the mainstream and becomes a major issue? Will this progress just as the old SMTP Relay issue did? Should we wait until it becomes an issue at which time the major ISPs will start implementing techniques that block requests from those servers that are open for DNS Recursion?
Am I, including the folks over at the DNS Report being paranoid? Or, are we just being proactive in addressing an issue that could potentially cause major harm. Don't just think about DNS here, think about your livelihood, your client's livelihood, their websites and email which are all controlled through DNS.
This takes me off guard. Last year I ran all the domains I manage through the DNS report and we got a clean bill of health. Your post prompted me to re-do the test and it report says we have this problem.
Thanks for the heads up. We will be bugging our hosting service about the report.
|Thanks for the heads up. We will be bugging our hosting service about the report. |
I can kind of guess what the response is going to be...
"It's been that way for years. Why all of a sudden is it now a problem?"
I guess it's going to take a bit of education along with a few large scale DDDoS Attacks to make this something that hosting providers take notice of.
From the folks at the DNS Report...
|We had been considering for years adding a warning about open DNS servers. There are a number of reasons why open DNS servers can be troublesome (many older open DNS servers were subject to cache poisoning, for example). However, now that there are attacks actually using open DNS servers, and since they are effective, and could cause serious problems for the owners/users of the DNS servers, we felt that it was important to add the test. |
Most non-administrators don't interact directly with DNS servers. So it's easy to lose sight of the two distinct functions a DNS server can perform.
A DNS server can advertise to the world the various resource records associated with your domain. This is known as "authoritative name service" and is probably only a familiar function to webmasters--in order to have a site, you need for some name servers to be authoritative for the hostnames associated with the site(s) in some domain. Generally your hosting provider operates these nameservers, and webmasters associate domains with these name servers at the time of domain registration (or at least with the registrar at some point in time after that).
The second function is completely separate. A "caching" or "recursive" DNS server can tell clients the answers to queries for various resource records, even records it can't resolve directly. Queries for non-local records are forwarded or recursed to servers that can answer them, and hte user is generally not aware of the handoff. All Internet users, even non-site-owners, use this service to resolve names to IP addresses.
Generally, the two services don't need to be accomplished by the same box. It's a good design to keep them separate. The BIND4 ops guide hints that you might want to separate these for performance reasons, with no discussion of the security benefits of separating them. The BIND8 docs don't really talk about it much beyond showing you how to disable recursion. The BIND9 Administrators Reference Manual suggests that you probably should separate these functions:
|The BIND name server can simultaneously act as a master for some zones, a slave for other zones, and as a caching (recursive) server for a set of local clients. |
However, since the functions of authoritative name service and caching/recursive name service are logically separate, it is often advantageous to run them on separate server machines. A server that only provides authoritative name service (an authoritative-only server) can run with recursion disabled, improving reliability and security. A server that is not authoritative for any zones and only provides recursive service to local clients (a caching-only server) does not need to be reachable from the Internet at large and can be placed inside a firewall.
(source: BIND 9 Administrators Reference Manual, chapter 1)
It's really always been best operations practice to separate the two functions onto different servers. It's just that these days there's more of a security emphasis in the rationale for doing it. In fact, there's an advisory about this exact problem that dates from 1999, so this isn't exactly breaking news:
My take is that perhaps "WARN" would be more appropriate then "FAIL" in reporting this issue. However, in the legend, they state "Rows with a FAIL indicate a problem that in most cases really should be fixed." I can see that point of view. We've (DNS admins that is) known for a long time this isn't best practice. It really probably is time to fix it.
|We've (DNS admins that is) known for a long time this isn't best practice. It really probably is time to fix it. |
linear, thank you very much for your input, it is much appreciated.
Yes, the issue has been present for years and just recently started receiving attention due to a few attacks where DDDoS was used. Rather than let it reach large scale proportions, I think it's a great idea to alert DNS Administrators to this problem and other DNS problems that may be present on the DNS Report.
The ultimate scenario for that report is to have no red and no yellow. I was at that point until this recent change with the DNS Recursion. We are now preparing to make the necessary changes so that we are not part of the next DDDoS Attacks. ;)
This is usually an issue for those that are on bit higher level of tech experience with running websites. Which is why we probably see so few posts here. It is a problem, but if someone wants to do a ddos or spam - they will find another way - this is just one of the ways to do this.
I've just checked some major shared hosting and dedicated server providers as they are typically attacked with DDOS, spam, etc.
Of 23 checked. 7 failed this test.
Can I post who failed this (no url dropping)? A lot of people who view this will probably have some sites on those providers servers. Although as mentioned before it might not matter cause it's just one hole of many that do/can get exploited so it might not do any good.
Great post by linear BTW.
A reason why one might want to contact their host or check their own is if their IP gets on the blacklist due to spamming as bcc1234 said. If you're IP/DNS is on that list you can't send mail to tons of customers because ISP's/Hosts use these lists for their filters.
|PASS -- OK. Your DNS servers do not announce that they are open DNS servers. Although there is a slight chance that they really are open DNS servers, this is very unlikely. Open DNS servers increase the chances that of cache poisoning, can degrade performance of your DNS, and can cause your DNS servers to be used in an attack (so it is good that your DNS servers do not appear to be open DNS servers). |
I wasn't aware of this potential problem, but it's good to know the DNS service I use is ahead of the game. Thanks for the head's up!
|Generally your hosting provider operates these nameservers, and webmasters associate domains with these name servers at the time of domain registration |
Is that right? I'm sure I host dns records with the company I register domain names with , this is different from the server providers. I tell my dns host the server name and *they* update the dns record, not the hosting company. If you host dns with the hosting co it can cause problems when you need to switch hosts as you need to also change the IPS TAG. Is that right?
As webmasters, knowledge of the DNS and in particular its implementation in BIND or other software is not something that generally comes to the forefront. Postmasters and those responsible for Internet connections in companies or organizations large enough to have a dedicated Internet connection need to know about it in some detail.
The diags at dnsreport are really great. I'd suggest webmasters look at them the way a company might look at an external auditor. You expect your IT staff to comply with industry best practices, and you pay an auditor to confirm it. However, the auditor doesn't know everything about your business, and there are times where a risk may be aceptable to you even though the auditor red flags it.
Effectively, unless you run your own name server, you are outsourcing this function to your hosting provider. This kind of check is excellent in ensuring you get solid value for the hosting dollar.
With hosting having gotten so competitive, there's a lot of variation in the degree of expertise brought to bear in system adminstration. Anything that lets us "size up" the quality of a hosting provider is a welcome tool.
It's hard to overstate the importance of DNS to a site owner--without working authoritative name service for your domain, no requests will arrive at your web server, and no mail will arrive at your inboxes. But this piece of infrastructure gets very little attention, probably because it usually just works.
@aspdaddy: there are some wrinkles to doing things in the UK that make it a little different than I described above. What you describe sounds like the UKish scenario.
Last November I moved from some shared webspace towards a managed server of my own. So all this is far beyond my knowledge but sounds as if well within my scope of responsibility.
How can I check my server-configuration concerning DNS lookup? (Its apache) How can I test whether I have been victim of any attacs already?
I dumbly recall that DNS-reverse-lookup became a problem about a year ago, when all of a sudden all my responses to aol-customers were refused. If I remember correctly, DNS-reverse-lookup had to be ACTIVATED, otherwise the mail could not be delivered. But I maybe wrong on that.
Your domain name registrar should have a record of which name servers are configured for authoritative name service for your domains. The old-school way is to do a whois query, but the DNSreport diags will show you the information under the heading "NS records at parent servers." It would also be good to see the same values listed at "NS records at your nameservers" in the same report.
Being attacked would be difficult to discern, but if there are any significant gaps in your log files, DNS issues can be at fault (wheter intentionally or inadvertently). If DNS is b0rked, requests generally don't make it to the web server.
Quick question: I have one nameserver that I use for all my different domains. By closing it, or making sure it's not an open dns server, will that affect my sites that use the the same DNS server, because they have different names?
|I have one nameserver that I use for all my different domains. By closing it, or making sure it's not an open dns server, will that affect my sites that use the the same DNS server |
By "closing" it, you are just making it unavailable for outsiders for anything other than the zones it is setup to be authoritative for. It should not have any negative impact on your sites that it provides resolution for.
BTW, are you sure you are using only one nameserver for your domains? Probably you have a secondary dnshost elsehere/upstream that is setup to mirror/slave your zones? If not, you may want to arrange for one. :)
PageOneResults - thanks for this very useful discussion.
We've not run open after having an incident many years back, so we are okay. I suppose that I always presumed folks would be aware of and not let their DNS servers do recursion for the known universe. Apparently, I am quite in error.
One caveat, obviously you do need to allow recursion on the names you actually do support (e.g., your own domain names as defined in the zone files for your own DNS servers). As was already mentioned on this thread, DNS really accommodates two functions. Don't shut down recursion on your own names. What you should shut down is the ability for other networks to resolve names that are not in your zone files.
FWIW, those interested in learning more about this topic might want to look at [isc.org...] for more information on the most widely used DNS server implementation.
I passed but got a "Warn"
related to my name servers in the same "C" class
which seems to indicate the same geo location
how big of a problem is that?
Thanks for the info. I've got two DNS, dns1.whatever.com and dns2.whatever.com. I also recently purchased secondary dns service from dyndns, and just got it setup.
This is a great thread - one of my goals for 2006 is to 'bulletproof' my sites as much as possible, and this is something I never would have thought of!
AHhh... why can't someone just directly DOS your website, why do they need to DOS your DNS?
I have my domains hosted at pair and they are known for locking things down more than most providers and they have theirs open. I sent them an email about it.
|I passed but got a "Warn" |
related to my name servers in the same "C" class
which seems to indicate the same geo location
how big of a problem is that?
The best practice is having your two (or more) nameservers that provide authoritative name service for your domain on two different networks. That way, if a routing problem makes one unavailable, the other is likely to take up the slack without anyone noticing interruption.
For most of us, if your hosting company's net is unreachable, it probably affects name servers and web servers alike. Better hosting companies will be multihomed and probably not likely to become unreachable due to a routing problem.
So warning is probably the appropriate level. If I were paying top dollar for hosting, I'd raise the issue with the provider though. Compliance with this best practice might be a useful "barometer" of quality.
Well my host told me they know about it and are working on fixing it. They said that have a problem because some of their customers have the wrong primary nameserver listed at their registry. If they fix it that will take them down. I'm ok with the fact that they are working it.
|If I were paying top dollar for hosting, I'd raise the issue with the provider though. Compliance with this best practice might be a useful "barometer" of quality. |
I am paying top $ for that dedicated server
so I will bring that question up
Indeed a great thread, surprisingly enough why isn't that thread interstings the majority?
I am now splitting two separated “C” classes
Thanks to that thread!
I inquired with the host of my dedicated server after seeing Fail for the Open DNS Server item. They replied that if they restrict DNS queries, the DNS server can't be used to query any domains, except the ones hosted on my server. They did not see any critical issues with this.
I guess my main question is why does the hosting company view it to be important that my DNS server be able to query "any domains"? How would loss of this functionality affect the operations of my site (i.e., user access, search engine access, etc.)?
Any guidance on this is appreciated since DNS is not my strength.
|AHhh... why can't someone just directly DOS your website, why do they need to DOS your DNS? |
The issue here isn't that they are DDoS'ing anybody's DNS. They are DDoSing a target computer, that could be a DNS server, could be a website, or just some poor guy's home Internet connection.
In the attack, the computer sending the packets could send them directly to the target. But, they can only send as much data as the size of their upload bandwidth. By sending packets to an open DNS server (with a forged source IP, of the victim), they can send a 70 byte packet to your DNS server, and your DNS server will send a 500+ byte packet to the victim. With EDNS0, that can be 4,000+ bytes.
So with a dialup account, it would be possible to saturate a T1.
What they would typically do in order to provide recursive name resolution to the boxes on their network is set up a separate, cacheing-only name server. (of course that costs money; even if the server already exists, someone needs to provision and test the software.)
So in talking with your providers, asking them to "move the cacheing name server to different infrastructure" would be clearer than asking them to "disable recursion on ns1.example.com and ns2.example.com." They still aren't going to like it much, because every host on their network has a resolv.conf file (or equivalent registry settings) that would need to be modified to reflect the change.
If your host has a service mentality, then they should be agreeable. The rationale for separating the functions is to improve the service level (by being more resilient to DDoS) after all.
[edited by: linear at 1:46 pm (utc) on Mar. 15, 2006]
|I inquired with the host of my dedicated server after seeing Fail for the Open DNS Server item. They replied that if they restrict DNS queries, the DNS server can't be used to query any domains, except the ones hosted on my server. They did not see any critical issues with this. |
The critical issue is that the DNS server can be used in part of a DDoS attack. That alone, though, won't be incentive enough for many to fix the problem.
However, if the DNS server is used in a DDoS attack, it's quite possible that most/all the outgoing bandwidth of that server could get used up. If that happens to both DNS servers of the host, then  all of the domains that the DNS server is authoritative for will be nearly unreachable, and  anyone using the DNS server for recursion will get DNS timeouts (E-mail won't go out, etc.).
The easiest option for your host is for them to enable recursion only for their local network. That lets them have recursion for customers who need it, while requiring almost no work for them.
>>How would loss of this functionality affect the operations of my site (i.e., user access, search engine access, etc.)?
I pointed my hoster to this thread and they responded by adjusting the dns-server and to find out if it had any negative effects on the service.
Turns out that some mail-redirects I have to an @gmail.com adres are now bouncing, because the server can't resolve gmail.com anymore.
Why not? because the server sends dns-requests to his own dns-server, wich forwards them to other servers if neccesary. Since it isn't allowed to forward the requests anymore, it fails!
Solution? Configure the server to use an external dns-server if it needs to resolve some domain-name, instead of the dns-server wich is installed on the server.
| This 81 message thread spans 3 pages: 81 (  2 3 ) > > |