Welcome to WebmasterWorld Guest from 54.161.128.52

Forum Moderators: phranque

Message Too Old, No Replies

SPF? not just for sunburns?

Sender Policy Framework

     
2:45 am on Sep 14, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


I sent an email to a customer tonight, and it bounced back with this message:

SMTP error from remote mail server after DATA:
host smtp.****bleep****.net [*****IP address****]: 550 5.7.1 SPF unauthorized mail is prohibited.


I thought I took care of this a long time ago. I don't understand the syntax, but my best advisors told me to go into WHM and create a DNS zone.

The record looks like:

example.com. (yes there's a dot after the TLD. Is that normal?)
14400 (I think that's the TTL)
IN (what does that mean?)
TXT (as opposed to CNAME or MX or A)
"v=spf1 a mx -all"

Is this correct?
3:15 am on Sept 14, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


ok it only took a few minutes on wikipedia to get a handle on the "v=..." part.

That SPF record says that mail is allowed to be sent from the "A" and "MX" systems, and all others should be forbidden.

So does this mean that the IP of the "sender's address" doesn't match the IP in the A record? And that's why the message is bouncing?

I'm sending my mail through the SMTP defined as "mail.(mydomain).com", authenticated by password.

The DNS record for the "mail" subdomain is a CNAME to (mydomain).com.

The MX record for (mydomain).com is [0] (mydomain).com.

The A record for (mydomain).com is the IP address of my server.

As far as I can tell, everything's set up properly. And... it's worked fine until just a few days ago when I started getting occasional messages bouncing back.
3:26 am on Sept 14, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


A test

As suggested by the SPF tools page:
[openspf.org...]

I sent an email to spf-test@openspf.org

The result: it bounced back (as it's designed to do), but the news wasn't good


This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

spf-test@openspf.org
SMTP error from remote mail server after RCPT TO:<spf-test@openspf.org>:
host mailout02.controlledmail.com [72.81.252.18]:
550 5.7.1 <spf-test@openspf.org>: Recipient address rejected:
SPF Tests: Mail-From Result="fail": Mail From="**bleep**@***bleep***.com" HELO name="vps.***myVPSdomain***.com" HELO Result="none" Remote IP="***bleep***"


What's the problem?
3:37 am on Sept 14, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


the problem is that the Remote IP - as echoed by the response from openspf - doesn't match the IP defined as the "A" record of my domain.

They look very similar. But the last octet is slightly different. Definitely something funky going on with my host.
3:50 am on Sept 14, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13266
votes: 363


What's the problem?

"bleep" seems to sum it up ;)

The sender is not authorized to send to the destination. This can be the result of per-host or per-recipient filtering.

But I like the generic x.7.x explanation better:
The security or policy status codes report failures involving policies such as per-recipient or per-host filtering and cryptographic operations. Security and policy status issues are assumed to be under the control of either or both the sender and recipient. Both the sender and recipient must permit the exchange of messages and arrange the exchange of necessary keys and certificates for cryptographic operations.

In other words, you need to get together with the recipient-- by carrier pigeon, I guess-- and agree on, uhm, something. Seems a bit extreme if all you were trying to say was "Yes, we have that in stock".
1:26 pm on Sept 14, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


Actually the mail in question is mission-critical for the normal functioning of my app, and the SPF dysfunction has worse consequences - I noticed that some hosts have added my domain to their spam blacklist. So, this not a trivial issue.

Thankfully, I've solved the problem and it wasn't difficult to do.

I'm running this site on a VPS, which hosts several domains. The root IP for the VPS is A.B.C.X (for example)

this one site in question has a dedicated IP, A.B.C.Y (for example)

So, the "A" record for the domain is A.B.C.Y, but the mail service is all hosted on the VPS, which is A.B.C.X. They don't match, and that's why the SPF Test failed.

My SPF record states "v=spf1 a mx -all" , meaning the IP of the sender-host must match the IP in the "A" record. It didn't.

Rather than fuss with the IPs, the solution was far simpler. I changed the SPF record to:

"v=spf1 a mx ip4:A.B.C.X -all"

Thus allowing the root VPS mail service to send mail originating from my domain.

Wait an hour for the DNS to propagate... then I sent another test message to spf-test@openspf.org

the response:


spf-test@openspf.org
SMTP error from remote mail server after RCPT TO:<spf-test@openspf.org>:
host mailout02.controlledmail.com [72.81.252.18]:
550 5.7.1 <spf-test@openspf.org>: Recipient address rejected:
SPF Tests: Mail-From Result="pass": Mail From="***bleep***@***bleep***.com" HELO name="vps.***myVPSdomain.com" HELO Result="none" Remote IP="***bleep***"


Now the SPF test passes.

SPF is an arcane art; I can appreciate how easy it is to ignore, or configure badly. But now I know the importance of doing it, and doing it right. The first time around I was merely pasting in an ignorant suggestion, and without knowing it my SPF test has been failing for a long time. It took me a couple of hours to read a little deeper, understand how SPF works, how to test if it is working, and figure out a solution.

I hope this thread helps someone overcome the same problems.
1:47 pm on Sept 14, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 2002
posts:742
votes: 8


As an email admin by day and webmaster by night I vote that this thread be placed in the WW Library.
2:29 pm on Sept 14, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


btw, the tongue-in-cheek title of this thread alludes to a more common meaning for the acronym SPF, "Sun Protection Factor", which is a number they stamp on sunscreen and lotion products in North America. Readers unfamiliar with that other meaning of SPF might be puzzled by that.
8:35 am on Sept 15, 2011 (gmt 0)

Full Member

5+ Year Member

joined:Jan 4, 2007
posts:221
votes: 0


Actually the mail in question is mission-critical for the normal functioning of my app


Do you have DomainKeys set up?

I recently added a forum to one of my domains (mostly to handle user registrations for tools on the site rather than for the forum itself) but it soon became clear that despite correct SPF and reverse DNS having been setup, certain email providers (including Yahoo) were bouncing mails from the server. Investigation indicated that lack of DomainKeys was the issue.

After a few searches I decided life was too short to include setting up DomainKeys on my current server and went with a third party, self-service, cloud based emailer ($0.001 per email - at my volumes $20 will last all year, maybe more :( ). Note if you're using PHP then the PEAR Mail package allows sending through 3rd party SMTP servers.
5:16 pm on Sept 15, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


No I don't have DomainKeys set up.

From the looks of their site, I don't think I'll be setting it up in this decade, either.

Look in the docs, under the heading "Implementation":

The signer needs to add code in the appropriate agent, to perform signing, and they need to modify their DNS administrative tools to permit creation of DKIM key records.

A validator needs to add code to the appropriate agent and then feed the result into the portion of their system needing it, such as a filtering engine.

The mere existence of a valid signature does not imply that the mail is acceptable, such as for delivery. Acceptability requires an assessment phase. Hence the result of signature validation must be fed into a vetting mechanism that is part of the validator's filter.


Seriously, that's it. The whole chapter. That's all you get.

DKIM = Yet another pile of esoteric technology that no one is going to use
8:48 am on Sept 16, 2011 (gmt 0)

Full Member

5+ Year Member

joined:Jan 4, 2007
posts:221
votes: 0


DKIM = Yet another pile of esoteric technology that I had to use (in some way) if I wanted people with yahoo email address to be able to register to use my tools!

Actually, my sunconcious has obviously been mulling this over for the last day I've got vague memories of the initial bouncing being related to switching forum software.

I emailed all currently active users ( I think it was approx 30 at the time!) that I was changing software, that the tools would be down on such and such a date, and if they couldn't log in when they were back up to let me know.

With the way my server was set up at the time I think it took only 6(!) identical emails to Yahoo addresses without DomainKeys for them to blacklist my web server as an email sender!
3:00 pm on Sept 16, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


Take heed: this SPF stuff requires that you change the DNS records for your domain. If you're familiar with DNS and how it works and how to change it without breaking everything, then all is well. But if you're new to DNS shenanigans, it's good to do a little reading & refresher before fiddling with it.

The SPF record has a type "TXT". Some of my servers are running WHM - and with that I'm able to add a "TXT" records to the DNS, no problem.

But I also have a couple of Rackspace Cloud instances, and using their management tool you can add an "A", "CNAME", "MX", but... not a "TXT". To add an SPF record to a Rackspace VPS, you have to open a support ticket and get one of their support staff to do it for you.
3:09 pm on Sept 16, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


My app sends notification emails to buyers giving them instructions for completing a sale. So, that's mission-critical, and when many people buy the same product the emails are pretty much identical. This may be more crucial than I thought.

To set up DomainKeys it's another "TXT" record added to the DNS.

Here's what an online tool generated for me, I have no idea yet if it's correct:

/._domainkey.example.com IN TXT "v=DKIM1; p={public key goes here}; s=email; t=y"
5:08 pm on Sept 16, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Jan 30, 2006
posts:1669
votes: 10



(yes there's a dot after the TLD. Is that normal?)


yes always
12:30 am on Sept 17, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 2002
posts:742
votes: 8


Two point worth mentioning for those on cPanel hosts:

The sending email server *may* be bound to a different IP than the webserver. If so add an additional ip4 server to setup.

Adding the DK/Sender ID/SPF records is easy - all forms and buttons.
12:40 am on Sept 17, 2011 (gmt 0)

Preferred Member

5+ Year Member

joined:Nov 20, 2007
posts:585
votes: 0


I recently had the same issue, mails getting bounced a lot, and ended up learning about SPF, then applying a similar rule to my dns which I must admit has helped greatly.

SPF has been around for sometime, but it does appear it is starting to be used quite heavily now.

I still have a problem with aol though, they seem quite happy in blocking too much. I have quiet a few important emails lost in the aol black hole, even though I'm not on any public blacklists.
12:56 am on Sept 17, 2011 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 2002
posts:742
votes: 8


AOL has a surprisingly good help page [postmaster.aol.com ] visit it and apply for whitelisting to fix your blocking issue.
4:54 am on Sept 18, 2011 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14650
votes: 94


I usually use the SPF Wizard on the MS site, works real nice
[microsoft.com...]
7:55 pm on Sept 22, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


that address for checking the validity of SPF (spf-test@openspf.org) was really handy. Without it I wouldn't have known for sure that everything was OK.

Is there such a thing for testing a Domain Key?
9:01 pm on Sept 22, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


Um I went into CPANEL and there's a button that said "Enable Domain Keys" and I clicked it. then the page refreshed and now it says "Status: Enabled & Active (DNS Check Passed)"

That's dandy. I don't know if it's really working. How can I check?

And then I'll need to do the same thing to some servers that don't have CPANEL. It'll be good for my soul to learn how to configure this from scratch.

Any tips for getting Domain Keys working on a Fedora box running postfix?
6:29 am on Sept 23, 2011 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 27, 2002
posts:765
votes: 0


I don't know if it's really working. How can I check?

Send yourself an email to a freemail account and check the raw headers. You should find a line starting with: DKIM-Signature
8:08 pm on Sept 23, 2011 (gmt 0)

Moderator from CA 

WebmasterWorld Administrator httpwebwitch is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 29, 2003
posts:4059
votes: 0


an email from the account in question, to my hotmail address.

after bleeping out personal details:


Authentication-Results: hotmail.com; sender-id=temperror (sender IP is --.--.---.---) header.from=------@-------.com; dkim=none header.d=------.com; x-hmca=none

X-Message-Status: n:0:n

X-SID-PRA: ----- ----- <-----@-----.com>

X-DKIM-Result: None

X-AUTH-Result: NONE

X-Message-Delivery: Vj0xLjE7dXM9MDws5TAhYT01O0Q9MTtTQ0w9Mw==

X-Message-Info: JGTYoYF78jGzuoPRzxQ33HYff5bTZMxB31RtcDy/c5MVSw1+ufv+rbWCLt7lLv+uM4fTNhHx0vpvpta+
MNDYBtFGrcpxjP9W1IIDSsU4EfvDFZsOucGHRBtjyQKWBecU5xomFn8cxbI=

Received: from ---.-------.com ([--.--.---.---]) by snt0-mc4-f12.Snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);

Fri, 23 Sep 2011 13:03:48 -0700

Received: from --------.--------.--------------.net ([--.---.--.--] helo=[10.10.10.217])

by ---.-------.com with esmtpa (Exim 4.69)

(envelope-from <-----@------.com>)

id 1R7ByI-0003gg-Sb

for -----------@hotmail.com; Fri, 23 Sep 2011 13:03:43 -0700

From: --- ---- <-----@-------.com>

Content-Type: text/plain; charset=us-ascii

Content-Transfer-Encoding: 7bit

Subject: testing DKIM

Date: Fri, 23 Sep 2011 16:03:32 -0400

Message-Id: <92E7E47D-E0AD-4145-9011-4AC22D448D5F@scubbly.com>

To: --- ---- <----------@hotmail.com>

Mime-Version: 1.0 (Apple Message framework v1082)

X-Mailer: Apple Mail (2.1082)

X-AntiAbuse: This header was added to track abuse, please include it with any abuse report

X-AntiAbuse: Primary Hostname - ---.-------.com

X-AntiAbuse: Original Domain - hotmail.com

X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]

X-AntiAbuse: Sender Address Domain - -------.com

Return-Path: -----@-------.com

X-OriginalArrivalTime: 23 Sep 2011 20:03:48.0470 (UTC) FILETIME=[E8326160:01CC7A2B]

testing!

[edited by: phranque at 12:54 pm (utc) on Sep 24, 2011]
[edit reason] fix thread width [/edit]