G-Mail has become Spam Haven

Report on recent forum experience + suggestion to help fix it

[AlexK]

G-Mail (gmail.com) email addresses used to be blanket-banned on the forums on my site. A week ago I un-banned them, together with mail.ru + a couple of others. Within an hour a G-Mail user spammed the forums. G-Mail is now banned tomorrow. Here's a brief report, plus suggestion on how to help fix it.

The bottom line from the last week:

• 1 in every 10 G-Mail users spams my forums
  (1 in 25 for all other users)
  
• Only 53% of G-Mail users register for a valid purpose
  (74% for all other users)
  

(mail.ru users are also heavy spammers, but way below G-Mail. If mail.ru users are accounted for, the site goes back to a `normal' 85% valid registrations.)

Custom routines on my site help to thwart bot-postings. The remaining trouble (and source for the stats above) comes from classic hand-crafted spam.

I want to allow G-Mail users to register on my sites, but the time + trouble they represent means that I cannot.

A suggestion to help fix these problems:
Google already offers methods within it's `Webmaster Tools' for registration of 'sitemaps', together with a python-script to produce them. `Sitemaps' themselves are XML-files. The XML format is ideal for the kind of automation that Google likes.

The suggestion is for a URL to which a registered webmaster can post XML reports of forum spam from G-Mail users.

The above para does assume that G will also act on the reports, of course. If a sufficient number of Webmasters pick it up, and if G puts the necessary systems in place, G-Mail could very rapidly become clean & spammers would go elsewhere.

I'm willing to build a PHP Class to help automate those reports, and could put it together in a day or two once a schema is decided. Many others in other languages (or the same!) would rapidly appear once endorsed by the major free-mail hosts.

What about it, Google?

[vordmeister]

For me one in two gmail addresses are spam registrations. That makes them easy to spot. Currently I look through new registrations and pre-moderate anyone using gmail together with another odd entry (ie location entered as US when their IP resolves to India).

I'd love to block them altogether but I have a couple of fantastic members who use gmail addresses. No such worries about anything.ru.

If google got their act together the spammers (and all the off-shore SEOs and social media promotion types) would just move elsewhere. I would prefer systems implemented by forum software providers that allow different treatment of new registrations based on what they submit during registration (including the email domain). I'd premoderate all gmail users if I could.

I'm anti-spam, but would be unlikely to help you send info to Google. Apart from anything else they already have too much info and their application system must be so broken we'd only be applying sticking plasters to it.

[AlexK]

vordmeister:

(I'm) unlikely to ... send info to Google ... they already have too much info

I'm also cautious about the amount of info they get. However, they can have as much info about spammers on my forums as they like.

I would integrate it into my ban routines. Just think how rewarding it would then be to press the 'Delete' button, knowing that the spammer was not just disappearing from your forum, but from the world.

If google got their act together the spammers ... would just move elsewhere

...and any other mail supplier that did not opt in to the system would eventually become a pariah. Perfect.

[vordmeister]

Just you and me in here then. It's a good point that shouldn't go unanswered.

My methods for validating posts as spam are:
1/ Does the post look doubtful?
2/ Are they using gmail?
3/ Where are they from?
4/ Then some more time consuming methods which shouldn't be public.

Many other forum admin just block gmail registrations. I'd be keen on the forum software people allowing me to chose a pre-moderated usergroup for new registrations based on conditions I set (such as email and location). Advantage of that is most out of the box forum operators wouldn't set the option so the spammers wouldn't need to get around it.

Some feedback for the mail providers would certainly be a good addition.

[AlexK]

vordmeister:

most out-of-the-box forum operators wouldn't set the option

Yes! That's the key that keeps most spam at bay on my site.

Google wants automated methods to handle most of it's business (makes good sense to me) backed up by human intervention for the remainder. The XML format is the means to that end.

The key issue is whether anyone actually cares - both Google & webmasters. It is a curious situation. If the question is put directly: "Do you want to help stop spam?", few are likely to say 'No'. The evidence says otherwise: I cannot find anywhere on G-Mail to report abuse of their system (please tell me if I've missed it).

An automated report system would be easy to setup and trivial to operate once in place. Google's system could easily collate reports of spamming from multiple sources and assign confidence, then act on it.

My methods for validating posts as spam are...

I operate a site concerned with Modems. My spam-algorithm is rather simpler...
1/ Is the post to do with Modems?
2/ There is no question #2.

[thecoalman]

I'd be keen on the forum software people allowing me to chose a pre-moderated usergroup for new registrations based on conditions...

phpbb3 has this feature but at the moment the only condition is post count. Under post settings you can set how many posts they need before they no longer need approval. Not sure if making other conditions was suggested or being considered but I'll forward the idea to the developers when the topic comes up.

[enigma1]

Does your forum s/w validates the account registration? Now usually these fake new accounts aren't setup by humans. This is a key point because you can simply deploy some form validation to check if the form is automatically submitted or not. That should get rid of the spam majority for the fake accounts.

Then when someone creates a new post the form can be verified again. Just in case human opens new account and then deploys a bot to spam.

One other area to check is where the visitors are coming from. Using ip/rnds and vice versa you can filter out non-ISPs because you can tell 1) if they resolve 2) if it's an ISP. For instance I don't see much of a point allowing visitors to post via known proxies.

[thecoalman]

Does your forum s/w validates the account registration?

There's a variety of form validations done. The trouble with any mass produced software is the "sameness". Since the people controlling the bots know what there up against they can program it to get around around speed bumps you put in place. 3.0.6 is going to have a captcha plug-in system which should help quite a bit in that regards because there will be a variety of captcha's.

One other area to check is where the visitors are coming from. Using ip/rnds and vice versa you can filter out non-ISPs because you can tell 1) if they resolve 2) if it's an ISP. For instance I don't see much of a point allowing visitors to post via known proxies.

You can block IP's or even use spam IP block lists for both registration and posting but I find they have too many false positives. In the end blocking via IP really doesn't work that well. There's too many holes for the bad guys to get through and you end up blocking legitimate users.

[enigma1]

you don't need to deploy captchas and make registration difficult. You can always validate forms using just css and html. The form will still look the same but say instead of a single submit button, you have say 100 buttons, 99 of which are hidden via css. The form script will know which one is the right one while a bot will have to decode the stylesheet something not practical. This can be dynamic too for example day/time and ip can be used to generate a different stylesheet for the form.

Also you don't have to block IPs. If you do rdns you can retrieve the DNS records. From there you can tell the target. So if the target say its verizon, comcast etc, ok its an ISP. But if say is goddady or rackspace, then is unlikely to have a human behind, probably a proxy or a compromised server. It won't eliminate every spam attempt but will help and you do it once per IP, then store the info.

Them a simple checkup for the HTTP headers. Say your forum is only in english, but someone attempts to register and the HTTP header for the language does not include "en", is a red flag. You need to take care of the spiders too as you need to place a whitelist in these cases.

From tests I have done with such countermeasures I do not get any spam attempts at all in the forms. They are plenty of attempts to enter a site but that's it. And there can be different levels. So you could allow everyone to browse but if they don't match the criteria to register you don't start sessions and you treat them like spiders. So no blocking of IPs in other words.

[vordmeister]

I like your ideas a lot. Bunch of stuff I've never thought about.

Problem for me with modding existing forum software is it gets updated too often so mods need to be re-written often. Generally robot registrations can be prevented using the software with a bit of fiddling, but IP stuff would be best done using the server.

Opening post wondered whether we could report emails from spam registrations. I think that would help a lot both for the bots and for the poorly paid humans that you still get after dealing with the bots. Will suggest that on a forum software board.

[ogletree]

The problem with gmail is that there is a very large base of legit users. I only use gmail.

[ken_b]

When was Gmail ever NOT a Spam Havem? Maybe the first 12 minutes of it's existence?

[thecoalman]

You can always validate forms using just css and html.

What if your registrant is seeing impaired and using a text reader? ;)

One thing to keep in mind is phpBB and other software like it needs to work on many different server configurations. That limits what you can do, many hosts don't allow outgoing connections to begin with.

[enigma1]

many hosts don't allow outgoing connections

You don't need to do outgoing connections at the phpbb level. Now if the host is blocking rdns for some strange reason (as the commands are part of the core php) then its time to change hosts.

What if your registrant is seeing impaired and using a text reader

Depends on the text reader. If it's reading what the eye sees, should be able to go through the forms. But if it's reading HTML then that's a problem with the s/w.