Trying to get /sendmail emails delivered to Gmail

[csdude55]

I have a form on my site for the user to request a forgotten password to be emailed to them. It's processed in Perl (sendmail) and sends an email to them.

The problem I'm having is that Gmail (and presumably Yahoo) always sends that email to the recipient's spam folder! And 99 times out of 100 the user doesn't understand to look there.

I have SPF, DMARC, and DKIM in the DNS. Gmail's headers show the SPF and DMARC, but I don't see any reference to DKIM in the header. This is from Gmail:

Authentication-Results: mx.google.com;
    spf=pass (google.com: domain of srs0=3dc0=hr=my.server.com=[e-mail] designates 123.45.67.89 as permitted sender) smtp.mailfrom="SRS0=3dc0=hr=my.server.com=[e-mail]";
    dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=mysite.com

The Perl script looks like this:

open (MAIL, "|/usr/sbin/sendmail -t") || die "Can't open /usr/sbin/sendmail!\n";

 # $found_email is set and verified earlier
 print MAIL "To: $found_email\n";
 print MAIL "From: info\@mysite.com (Me)\n";
 print MAIL "Subject: Blah blah blah\n\n\n";

 print MAIL "yada yada yada";
close (MAIL);

Is there a trick to getting sendmail to send the DKIM header?

[engine]

I'd be interested to know solutions as I've seen similar issues. The most successful solution I've found to date is not technical, but is simply to put clear instructions on the form to check spam folder.

[csdude55]

Well, I can't find a way to authenticate with sendmail, so I've spent today rebuilding using Net::SMTP. I had to install Net::SMTP, IO::Socket::SSL, and MIME::Lite to fit the example I found:

[skipser.com...]

And here's my final test script:

use Net::SMTP;
use MIME::Lite;

$login_user = '[e-mail]';
$login_pass = <PASSWORD>;

$to = '[e-mail]';

$msg = MIME::Lite ->new (
From => '[e-mail]',
To => $to,
Type => 'text/html',
Subject => 'Testing Net::SMTP',

Data => qq~
The <i>body</i> of the <b>email</b> Net-SMTP example
~
);

$smtp = Net::SMTP->new(
'mail.mysite.com',
Port => 465,
SSL => 1,
Debug => 1,
);

$smtp->auth($login_user, $login_pass) or die('Could not authenticate');

$smtp->mail($login_user);
if ($smtp->to($to)) {
$smtp->data();
$smtp->datasend( $msg->as_string() );
$smtp->dataend();
}

else {
print "Error: ", $smtp->message();
}

$smtp->quit;

print "Done";
exit;

The email successfully went through... but it still went to Gmail's spam folder :-/

When I "Show original", though, it does show DKIM:

Message ID<[e-mail]>
Created at:Wed, Dec 6, 2023 at 4:27 PM (Delivered after 2 seconds)
From:[e-mail] Using MIME::Lite 3.033 (F2.84; T2.24; A2.21; B3.16; Q3.16)
To:[e-mail]
Subject:Testing Net::SMTP
SPF:PASS with IP 123.45.67.89 Learn more
DKIM:'PASS' with domain mysite.com Learn more
DMARC:'PASS' Learn more

WHM's Delivery Report shows that the email has a spam score of 2, so maybe that's why it was filtered? I honestly don't know.

The most successful solution I've found to date is not technical, but is simply to put clear instructions on the form to check spam folder.

I have that, too, but I'm pretty sure that my users can't read! LOL I have AT LEAST 2 emails a day from users that requested the password and didn't get it in their Inbox. The wide majority of the time, it was in their spam folder :-/

[haramamba]

Did you check the ip address on spamhaus?
Two years ago Gmail decided to reject all my emails on a new ip address until I cleared it on spamhaus.
PS I have only SPF.

[engine]

>users can't read!
Often don't read. Been faced with that for decades!

Keep experimenting with the notification and prominence of the message for people to check their spam folder.

[mack]

I would say try and migrate your script to use SMTP if possible. I know this will be very dependent on your host and what they allow but it does solve a lot of email problems that tend to be common with both sendmail and phpmail (on the PHP side).

I had a similar problem with a number of outgoing emails.. some were for forgotten password reminders others were for email verification upon sign-up. Using SMTP certainly resulted in no bounces and a better chance of getting into the main inbox.

Mack.

[csdude55]

Did you check the ip address on spamhaus?

Yep, I get a regular report from MXToolbox on that.

I tested the email with mail-tester.com and it said that I had a perfect spam score. Then I tested with dkimvalidator.com, and it passes everything and says that my SpamAssassin score is -5.106. But I just tried again, and the exact email that the testers approved STILL went to Gmail's spam folder :-/

I've created a Message ID (way more work than anticipated!), but that didn't help. I also figured out to use MIME::Lite's scrub() to remove the X-Mailer and Content-Disposition tags, but that also didn't help.

My only remaining theory:

The POP account I'm sending from is [e-mail], and the From address is [e-mail]. Maybe Gmail doesn't like the noreply, or that they're not the same?

I would say try and migrate your script to use SMTP if possible.

My server uses Exim for regular emails, but the docs for Mail::Transport::Exim were super vague and I kept finding references to using Net::SMTP instead. Which I think is the same thing that you're suggesting? Or is there a better way to use SMTP?

[haramamba]

My dkimvalidator test results:

SpamAssassin Score: -5.01
Message is NOT marked as spam
Points breakdown: 
 0.0 RCVD_IN_ZEN_BLOCKED_OPENDNS RBL: ADMINISTRATOR NOTICE: The query
        to zen.spamhaus.org was blocked due to
       usage of an open resolver. See
       https://www.spamhaus.org/returnc/pub/
       [x.x.x.x listed in zen.spamhaus.org]
-5.0 RCVD_IN_DNSWL_HI  RBL: Sender listed at https://www.dnswl.org/,
       high trust
       [x.x.x.x listed in list.dnswl.org]
-0.0 SPF_HELO_PASS   SPF: HELO matches SPF record
-0.0 T_SCC_BODY_TEXT_LINE No description available.

I'm sending only plain text, no MIME. My headers:

Received: from localhost (localhost [127.0.0.1])
From:
To:
Subject:
Message-Id:
Date:

Hello User,
Some text

I'm using Postfix, it does not include any mailer-related info into the headers.

[csdude55]

Mine is similar. I was originally sending in plain text, too, but mail-tester took off a few 0.x because I didn't have HTML. So I modified the test to include HTML.

SpamAssassin Score: -5.106
Message is NOT marked as spam
Points breakdown: 
 0.0 RCVD_IN_ZEN_BLOCKED_OPENDNS RBL: ADMINISTRATOR NOTICE: The query
               to zen.spamhaus.org was blocked due to
              usage of an open resolver. See
              https://www.spamhaus.org/returnc/pub/
              [123.45.67.89 listed in zen.spamhaus.org]
 0.0 URIBL_DBL_BLOCKED_OPENDNS ADMINISTRATOR NOTICE: The query to
              dbl.spamhaus.org was blocked due to usage
              of an open resolver. See
              https://www.spamhaus.org/returnc/pub/
              [URIs: gonc.net]
 0.0 URIBL_ZEN_BLOCKED_OPENDNS ADMINISTRATOR NOTICE: The query to
              zen.spamhaus.org was blocked due to usage
              of an open resolver. See
              https://www.spamhaus.org/returnc/pub/
              [URIs: gonc.net]
-5.0 RCVD_IN_DNSWL_HI    RBL: Sender listed at https://www.dnswl.org/,
              high trust
              [123.45.67.89 listed in list.dnswl.org]
-0.0 SPF_HELO_PASS     SPF: HELO matches SPF record
 0.0 HTML_MESSAGE      BODY: HTML included in message
-0.1 DKIM_VALID       Message has at least one valid DKIM or DK signature
 0.1 DKIM_SIGNED      Message has a DKIM or DK signature, not necessarily
              valid
-0.1 DKIM_VALID_AU     Message has a valid DKIM or DK signature from
              author's domain
-0.0 T_SCC_BODY_TEXT_LINE  No description available.
 0.0 TVD_SPACE_RATIO    No description available.

For my test, the subject is Testing Net::SMTP and the body is:

This is a multi-part message in MIME format.

--_----------=_170201855399430
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain


This is the ID:
1702018553-7246cc4ded783e20548955f80f98dbc7-4165f4608534e7e15bb78aa956f6210=
[e-mail]

--_----------=_170201855399430
Content-Transfer-Encoding: 8bit
Content-Type: text/html


<body>
<b>ID:</b><br>
1702018553-7[e-mail]
</body>

--_----------=_170201855399430--

At this point I'm at a loss, I just don't know how to make it better than -5.106.

[not2easy]

Have you looked at gmail's requirements/specs for email? I know they changed something mid-2022 and it was suddenly quite a pain to get my gmail via ThunderBird email client. I realize the changes affected receiving, not sending, and the instructions I got were for ThunderBird client but here is what they required to receive email:

The correct servers are imap.gmail com and smtp.gmail.com, and the authentication method for both should be OAuth2. Check that cookies are allowed in TB Options/Privacy & Security, remove the Saved Passwords in Options/Privacy & Security, restart TB, enter the account password in the OAuth browser window when prompted.

For a long time after that (about 10 months, all my automated notifications sent from my sites were routed to the spam folder. I kept marking them as not spam and finally they stopped sending them to the spam folder.

Have you tried sending via IMAP instead of POP?