homepage Welcome to WebmasterWorld Guest from 54.211.190.232
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

    
Have you checked your DNS lately?
pageoneresults




msg:3693555
 6:46 pm on Jul 8, 2008 (gmt 0)

Hey, I'm guilty! Okay, so I failed to run a DNS Report on a few sites due to a "false sense of security". Yes, I too suffer the same challenges that many of us do when dealing with the alphabet soup of protocols, best practices, etc.

One of the areas that I got real lax in was "spot checking" DNS for websites that we do not control DNS for. A few years ago I went on this mission to get DNS locked down within our network so I could have a "sense of security" that if something ever were to go wrong, we couldn't blame DNS which seems to be the nemesis of many here at WebmasterWorld. And, many don't know it just yet. :(

Just yesterday, we ran a DNS report for a domain that we are launching a new application on. My programmer out of "best practice" ran the report and what did we find? Okay, you tell me, if you were in charge of DNS, what would this say to you?

NS
----------
FAIL - Open DNS Servers
WARN - All nameservers report identical NS records
FAIL - Missing nameservers 2
WARN - TCP Allowed

SOA
----------
FAIL - NS agreement on SOA Serial #

Mail
----------
WARN - Mail server host name in greeting
WARN - Acceptance of postmaster address
WARN - SPF record

Yes, I do believe email is sent from the same server network and the errors are across the board.

If you had 100+ websites under DNS which reported the above FAILs and WARNs, how would you feel?

As an Internet Marketer, the above would be a concern for me. Especially the Open DNS Server FAIL. I've been through that issue before and many brushed it off as the whole "small percentage" bit. That's fine but, does that mean that these holes should be left open for miscreants?

Some history on the Open DNS Servers challenge...

DNS Recursion - Open DNS Servers
The new open relay problem. Are you addressing this?
[webmasterworld.com...]

 

coopster




msg:3693613
 7:42 pm on Jul 8, 2008 (gmt 0)

DNS is certainly hosed up for the account you are handling. Yes, I know you did not set it up but you were wise to have a look at it. You might certainly eliminate some headaches right up front before you do any development!

The mail warnings are quite common as many people ignore the RFC standard when it comes to the "postmaster" and "abuse" email addresses for the mail server. Here the folks at dnsstuff are letting you know that the accounts have not been set up even though the RFC calls for it. It's just that so many people are tired of the spam they just decided not to set up the well-known mail accounts because they have been abused so much. SPF records are not a standard, so that is merely a warning too.

However, that first mail warning can create some big issues. Many servers, including ATT, may reject send mail attempts if the mail server does not have rDNS set up properly for your mail server(s). If they do a reverse DNS lookup on IP and find that the host name does not match they may reject the attempted mail delivery.

Demaestro




msg:3693621
 7:52 pm on Jul 8, 2008 (gmt 0)

If they do a reverse DNS lookup on IP and find that the host name does not match they may reject the attempted mail delivery.

This just happened for a new site of ours. Same thing as the thread starter.. a heads up employee checked the reverse DNS and found it wasn't set up properly.

Because I am a developer a rely on my sys guys for this stuff... it is good to know my guy is on top of these things... I wasn't really aware of the consequences even though I was aware of the protocol... I suspect that this is true for many.

pageoneresults




msg:3693660
 8:24 pm on Jul 8, 2008 (gmt 0)

DNS is certainly hosed up for the account you are handling.

lol, I'm going to quote you! :)

You might certainly eliminate some headaches right up front before you do any development!

It "was" the first thing on my list back when the management of DNS transferred to a Corporate "Entity".

The mail warnings are quite common as many people ignore the RFC standard when it comes to the "postmaster" and "abuse" email addresses for the mail server.

I'm not going to let this one get brushed aside as "quite common". While they "may be" quite common, does that mean they should be ignored? Have you looked at the SpamAssassin "default" rules out of the box for these two particular issues? Does "quite common" mean that providers should be RFC Ignorant?

header Envelope sender in postmaster.rfc-ignorant.org DNS_FROM_RFC_POST 0 1.376 0 1.614
header Envelope sender in abuse.rfc-ignorant.org DNS_FROM_RFC_ABUSE 0 0.374 0 0

The Apache SpamAssassin Project
Tests Performed: v3.0.x
[spamassassin.apache.org...]

Here the folks at dnsstuff are letting you know that the accounts have not been set up even though the RFC calls for it. It's just that so many people are tired of the spam they just decided not to set up the well-known mail accounts because they have been abused so much. SPF records are not a standard, so that is merely a warning too.

Oh-oh "Danger Will Robinson, Danger"! SPF Records are a standard. If they were not, why are the major spam prevention software providers using SPF in their scoring algorithms? Check this out, these are the "default" scoring mechanisms used by SpamAssassin, there are "only" seven (07) of them in the tests...

header SPF: sender matches SPF record SPF_PASS -0.001
header SPF: sender matches SPF record SPF_PASS -0.001
header SPF: sender does not match SPF record (fail) SPF_FAIL 0 0.001 0 0.875
header SPF: sender does not match SPF record (softfail) SPF_SOFTFAIL 0.500 0.842 0.500 0.500
header SPF: HELO matches SPF record SPF_HELO_PASS -0.001
header SPF: HELO does not match SPF record (fail) SPF_HELO_FAIL 0 0.405 0 0.001
header SPF: HELO does not match SPF record (softfail) SPF_HELO_SOFTFAIL 0 1.002 0 3.140

However, that first mail warning can create some big issues. Many servers, including ATT, may reject send mail attempts if the mail server does not have rDNS set up properly for your mail server(s). If they do a reverse DNS lookup on IP and find that the host name does not match they may reject the attempted mail delivery.

Thank you for explaining that one to those reading. I'm not even going to cut and paste the scoring used on hostname challenges. That is a rather large one and I wish they would mark it as FAIL instead of WARN.

Because I am a developer a rely on my sys guys for this stuff... it is good to know my guy is on top of these things... I wasn't really aware of the consequences even though I was aware of the protocol... I suspect that this is true for many.

You are fortunate. I think the majority of our readers here may not be as fortunate. I literally used to be one of them. It has taken me "years and years" to finally understand where to begin looking.

Also, let's get some feedback on that first group...

NS
----------
[b]FAIL[/b] - Open DNS Servers
[b]WARN[/b] - All nameservers report identical NS records
[b]FAIL[/b] - Missing nameservers 2
[b]WARN[/b] - TCP Allowed

MichelleStute




msg:3693730
 9:28 pm on Jul 8, 2008 (gmt 0)

I was just involved with a similar situation as what P1R described. Everything is controlled by 'Corporate,' but luckily, someone on my web team ran a DNS report and we were alerted to some big problems.

I'm on the internet marketing side of things, and am by no means a techie. So I am often the go-between who translates what the techies tell me to my bosses. How do you put this stuff into layman's terms for your clients?

BTW, our tech team had this issue on their to-do list but it was not top priority because our sites are mostly brochure sites. They apparently weren't as concerned about the ramifications for email.

coopster




msg:3695077
 2:46 am on Jul 10, 2008 (gmt 0)

I'm not going to let this one get brushed aside as "quite common". While they "may be" quite common, does that mean they should be ignored?

I never said they should be ignored. I merely stated the fact that they are quite often overlooked when a user sets up their site. It may be for the reason I stated, or even more likely it is because the average person setting up a web site using a shared hosting provider has no idea the standard exists. So, yes, it can more often than not be attributed to lack of knowledge (ignorance).

SPF Records are a standard.

Show me. SpamAssassin et al provide many different ways for people to analyze mail. Because a rule can be written does not necessarily mean it will be employed. Nor does it mean it is a standard. Case in point, for clarification:

header Subject talks about losing pounds SUBJECT_DIET 2.527 1.621 2.084 1.466

Unless I missed that meeting, this is not a standard. SPF is not a standard.

BTW, the SpamAssassin test link is a couple releases behind. Here is the latest:
[spamassassin.apache.org...]

Also, let's get some feedback on that first group

As I first stated, this is where top priority should be right now. The first question I would ask is "Are they running their own DNS server(s) and if so, who is the person responsible for set up and maintenance?"

coopster




msg:3695085
 2:55 am on Jul 10, 2008 (gmt 0)

How do you put this stuff into layman's terms for your clients?

First, since they are your client, you just be open and honest with them regarding your concerns and the possible pitfalls awaiting them. You can't throw technical jargon at them or talk over their heads. Dumbing it down is an art. Explain a real-world scenario for them that is specific to their business, their culture, their expectations of what their site should be and how it should operate from day-to-day. Then explain how improper setup can negatively impact their business.

As their professional partner, you have an obligation to let them know. If you can do so without sounding like a salesman, you've done your job. In writing is often a good idea, too, by the way. That way, if they do not take action you have a gentle reminder to fall back on later if the case calls for it.

pageoneresults




msg:3695290
 10:44 am on Jul 10, 2008 (gmt 0)

I never said they should be ignored.

I know, I have to be careful and be more literal in my responses. But, when I read this...

The mail warnings are quite common...

That more or less gives everyone reading the impression that they are okay to ignore since they are "quite common", yes? ;)

Show me.

Ooops, busted! But, not really...

Lots of domains have published records, including AOL, Amazon, Google, O'Reilly, SAP, TicketMaster, Mail.com, w3.org, Earthlink and Verizon. And the ones who haven't published are working on it.

We expect adoption to pick up exponentially; according to some estimates, the number of sites checking SPF doubles every three weeks. SPF plugins are available for all the major Mail Transfer Agents (MTAs).

For me coopster, it is a Standard and has been since 2004 October. Back in the beginning when we were first "understanding" it all, there was an eye opening experience with "554 refused mailfrom because of SPF policy" bounced emails where we had some misconfiguration issues initially which were fixed immediately. Those bounces told me that hey, these recipient servers are checking for SPF and that was back in late 2004 early 2005. Fast forward 3 years and SPF is now a standard within various high level organizations like Financial Institutions, Government Agencies, Google, etc.

coopster, do you use SPF?

SPF Adoption Rates
[openspf.org...]

Because a rule can be written does not necessarily mean it will be employed.

When they are part of the "default" ruleset, are those the ones that are "usually employed"? By default? And then you would go in and change those if you so desired?

BTW, the SpamAssassin test link is a couple releases behind. Here is the latest: [spamassassin.apache.org...]

Ah, touche! Thank you for the updated link, I overlooked that in my Bookmarks. Shame on me! ;)

As I first stated, this is where top priority should be right now.

Ah, the meat of the matter.

The first question I would ask is "Are they running their own DNS server(s).

I do believe they are.

Oh, in regards to the SpamAssassin Rules, thank you for the updated link. That just gives me more ammunition for the discussion. Look at the difference between 3.0 and 3.2. They've beefed things up a bit and also added two new rules for SPF in 3.2. ;)

SpamAssassin Default Rules 3.0 vs 3.2

3.0 - header SPF: sender matches SPF record SPF_PASS -0.001
3.2 - header SPF: sender matches SPF record SPF_PASS -0.001

3.0 - header SPF: sender does not match SPF record (fail) SPF_FAIL 0 0.001 0 0.875
3.2 - header SPF: sender does not match SPF record (fail) SPF_FAIL 2.600 0.992 1.669 0.693

3.0 - header SPF: sender does not match SPF record (softfail) SPF_SOFTFAIL 0.500 0.842 0.500 0.500
3.2 - header SPF: sender does not match SPF record (softfail) SPF_SOFTFAIL 2.301 0.654 0.698 0.596

3.0 - header SPF: HELO matches SPF record SPF_HELO_PASS -0.001
3.2 - header SPF: HELO matches SPF record SPF_HELO_PASS -0.001

3.0 - header SPF: HELO does not match SPF record (fail) SPF_HELO_FAIL 0 0.405 0 0.001
3.2 - header SPF: HELO does not match SPF record (fail) SPF_HELO_FAIL 2.298 0.365 0.540 0.001

3.0 - header SPF: HELO does not match SPF record (softfail) SPF_HELO_SOFTFAIL 0 1.002 0 3.140
3.2 - header SPF: HELO does not match SPF record (softfail) SPF_HELO_SOFTFAIL 2.599 1.533 1.427 0.841

Two (2) new rules in 3.2

3.2 - header SPF: sender does not match SPF record (neutral) SPF_NEUTRAL 2.199 1.210 0.756 0.686
3.2 - header SPF: HELO does not match SPF record (neutral) SPF_HELO_NEUTRAL 2.231 2.000 0.744 0.576

BTW, our tech team had this issue on their to-do list but it was not top priority because our sites are mostly brochure sites. They apparently weren't as concerned about the ramifications for email.

Unfortunately that is how many view this whole DNS thing. Since it is really the responsibility of the person maintaining the servers, DNS, etc. "you" should not have to be concerned about it. Well, in the real world, I think everyone with a website should be concerned.

I just can't see leaving something like this up to fate. If there are FAILs and WARNs, why not just fix them and stop using the excuse that "it is quite common". Yes it is, and so is HTML tag soup but that doesn't mean we should perpetuate and/or prolong those "not best practices".

coopster




msg:3695468
 2:52 pm on Jul 10, 2008 (gmt 0)

coopster, do you use SPF?

No ... and yes. My personal opinion is somewhat neutral in regards to the effectiveness of SPF. It's not that I harbor any anti-SPF Records [webmasterworld.com] sentiment (follow the link in that Supporters thread to read an article by Jonathan de Boyne Pollard). However, if it ever gets closer to standards adoption, I would like to see some changes in the proposed accreditation process. I've harped about this before around here and rather than get on my soapbox I'll leave these comments and move on ... I have tested with and without SPF TXT records in DNS and have noticed no difference. If monitoring is not on at the server it won't make a difference anyway and since it is not a standard, many servers do not recognize/acknowledge SPF. Does it hurt to have the TXT record in DNS? Currently, I don't believe it hurts. Is it effective? Argumentative, at best. Is it a standard? ;) No, not at this time.

I can offer a couple of experiences here to explain ...

I was troubleshooting a new server for a customer back in January trying to figure out why mail was not getting delivered to certain domains via a PHP application. I did not do what you did here, I did not check DNS first. Stupidity on my part as I dug straight into the code. It was a hard lesson learned and also why I commend you for doing so first! I did check the MX records in the DNS first, but that was it (I did NOT check rDNS at this time -- ignorance on my part). All was well, good to go. After confirming the code worked fine I checked black lists, nothing there. I properly set up SPF, no change. It wasn't until I checked rDNS and requested the assigner of the IP to set up reverse delegation that I was able to put mail through. Since then I was contracted to do the same on another contract. DNS was the first thing I checked this time. The user had MX set up, SPF, everything. So I grabbed my little "dig" utility belt and started pinging the rDNS and found 3 mail servers without rDNS. Bingo.

Both accounts it was believed to be software-related issues, neither case had a clue that it may be DNS-related.

By the way, if your client will give you access to the DNS zone file management or at least a printout of the zone file(s) being used, you can cut your analysis time and save them money. Make sure they understand this. They may have multiple subdomains being used for sending mail and you won't know it until you have been banging your head against the wall for awhile or by chance you receive a test email from one of the servers and the headers inform you of their existence.

Properly configuring MX records and DNS (including rDNS) for a mail server are key to successful operation. That is an obvious statement, but not so obvious to "everyone with a website" that should be concerned! That level of knowledge is something that must be attained or hired and it usually doesn't happen until issues arise. Proper DNS configuration is (unfortunately) more a reaction than proactive.

"usually employed"

That just gives me more ammunition for the discussion.

SPF support requires a perl module that needs to be setup and activated before the ruleset is employed. In most cases, it is not. More info in the SA wiki:
[wiki.apache.org...]

pageoneresults




msg:3695612
 6:00 pm on Jul 10, 2008 (gmt 0)

I have tested with and without SPF TXT records in DNS and have noticed no difference.

Hmmm, I do believe I have a similar test with a "difference". And a very recent one, like in the past 60 days.

If monitoring is not on at the server it won't make a difference anyway and since it is not a standard, many servers do not recognize/acknowledge SPF.

Google does. Yahoo! does. Hotmail does. AOL does. Shall I continue to add to that list of top Brand names that are or have adopted SPF? ;)

BITS have also made it one of three (03) recommendations to the financial industry to be adopted by 2008 October. Many are going to ask who BITS are. BITS is a nonprofit industry consortium of 100 of the largest financial institutions in the U.S.

BITS Email Security Toolkit - Protocols and Recommendations for Reducing the Risks
[bitsinfo.org...]

The BITS Email Security Working Group recommends the adoption of three specific technologies to enhance email security:
• Transport Layer Security (TLS);
• Sender Authentication—Sender ID Framework (SIDF) or Sender Policy Framework(SPF)); and
• DomainKeys Identified Mail (DKIM)

Properly configuring MX records and DNS (including rDNS) for a mail server are key to successful operation. That is an obvious statement, but not so obvious to "everyone with a website" that should be concerned!

Obvious? That's an understatement! ;)

That level of knowledge is something that must be attained or hired and it usually doesn't happen until issues arise. Proper DNS configuration is (unfortunately) more a reaction than proactive.

It took me how many years to figure that out?! In the end, its now going back out to a third party provider who specializes in mail. It has gotten way too expensive to deal with email issues these days. There are too many customers in the "free email mindset" and a local provider of these services does it as a loss leader usually. I surely don't want to look at email as a loss leader, that would be an injustice, it still works for the time being.

And then we have the other side of things and that is dealing with the consumer of email services. It really isn't worth my time when I have someone calling me at 0300 in the morning because they decided to get up early and realized their mail wasn't working. Only to find out that they didn't have Internet access either, nor was their Cable TV working. If I were to look at the time investment in dealing with email, the majority of it is chasing consumer complaints that are typically black holes. Or, they lead back to a local PC issue outside of your control. That's a tough business model and I ain't given that away for free, no way!

SPF support requires a perl module that needs to be setup and activated before the ruleset is employed. In most cases, it is not.

It appears that SPF support was adopted by SpamAssassin with the release of 3.0, correct? That was in 2004-09-22 right before the official SPF Annoucement in 2004 October. They are now at Release 3.0.3 which came out in 2005-04-29. Am I to believe that the "in most cases" has not changed in four (4) years?

SpamAssassin Release History

3.0.3 2005-04-29
3.0.2 2004-12-16
3.0.1 2004-10-23
3.0.0 2004-09-22

My personal opinion is somewhat neutral in regards to the effectiveness of SPF.

It's not that I harbor any anti-SPF Records sentiment (follow the link in that Supporters thread to read an article by Jonathan de Boyne Pollard).

I read that article! Wow, makes me think twice about using SPF and now I know why you may be somewhat neutral. Heck, if you read enough of that, you'd begin to wonder if the world was going to end tomorrow. But, I read it again and then it makes me think twice about the author's intent. That article is more of a "rant" than anything else. It surely provides a laundry list of reasons not to use SPF. But I'm going to chalk that article up to a bit of "bagging" taking place between a small group of experts in this area. Each one of them have their reasons to promote particular protocols. Those guys/gals are at levels I can only aspire to in regards to understanding and being able to really "debate" all of this. ;)

Stupidity on my part as I dug straight into the code. It was a hard lesson learned.

Oh come on coopster, that's a bit harsh isn't it? Remember, these are quite common and it happens to all of us. :)

Okay, now that we've beat up one of six (06) areas within the DNS Report, can we discuss some of the others too? The one we've been beating up is the one most "blow off" anyway because they are quite common.

Oh, by the way, Japan just saw a major increase in SPF adoption...

Measurement Results on Deployment Ratio of Domain Authentications
[member.wide.ad.jp...]

coopster




msg:3695804
 10:27 pm on Jul 10, 2008 (gmt 0)

Hmmm, I do believe I have a similar test with a "difference". And a very recent one, like in the past 60 days.

hehe, if you are referring to the server mentioned in the OP of this discussion, I can tell you why, and it won't be SPF!


Google does. Yahoo! does. Hotmail does. AOL does. Shall I continue to add to that list of top Brand names that are or have adopted SPF? ;)

That's only four companies! And I openly admit there has been absolutely no correspondence issues with any of them since their adaptation, without any SPF records setup in DNS ;)

Am I to believe that the "in most cases" has not changed in four (4) years?

The SA software has been ready, but without that perl module we have the "in most cases" situation. As I said, details in the wiki docs referenced. Don't believe me? There are instructions on how to check -- go ahead and run the command on your mail server. See if you are actually using the SA SPF support. I'll bet most people will be surprised.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved