Frustrated with Spambots Coming In through Webmail Forms

[hermosa]

The problem seems to be getting worse. I have set up at least 20 filters, changed the e-mail address and still they come day after day. It seems that they are getting more aggressive the more I filter them out. Am I imagining it?

I have read about options to Matt's Script Archive nms or something like that but it's too WAY complicated. Perhaps something simpler could be suggested. I just don't understand it. It says something about .php I haven't got a clue how to do that. Do I have to create a file ending in .php? How do I set that up.

[tedster]

If there was an easy solution, the spam bot companies would be out of business! And no, robots.txt only is effective with well-behaved bots, and that certainly doesn't describe bots that create form spam. For robots.txt to work, the people who write the bot have to program in obedience to robots.txt.

PHP is a scripting langauge - a whole world, in fact, with its own forum here [webmasterworld.com]. Most spam prevention takes requires scripting on the server to stop the bulk of it from getting through. Subscribing to the various blackists and using that data can be a big help, but these guys definitely are persistent.

[Lord Majestic]

Use "Contact Us" form without publishing email address - use JavaScript that would print extra HTML variable that is necessary for your form submission to work - you can put some human message if JavaScript is not enabled in browser to warn legit users. This approach cut down spam on our BBS to zero (only had a few instances of manual spam in the last couple of years).

[tedster]

Another similar approach uses external css to HIDE an unneeded form input. Anytime that box is filled in, it's a spambot so you just throw the submission out.

[The Contractor]

Hopefully your forms "mailto/recipient" is not available in your HTML code/page? Recent discussion at [webmasterworld.com...]

[justgowithit]

Another similar approach uses external css to HIDE an unneeded form input. Anytime that box is filled in, it's a spambot so you just throw the submission out.

I've never tried this approach but it seems like it would could be very effective. Tedster, have you employed this method with success?

[hermosa]

Again, the explanations sound good but more detailed instructions are needed for the HTML/PHP challenged. What exactly do I put in what file and where:

What do I call the .php file and what goes in it?

What changes do I make to the code in the HTML page:

<tr>
<td align="left" colspan="3"><FORM METHOD=POST ACTION="/cgi-bin/formmail">
<H1 Align="center">Text</H1>
<H4 align="center">Text</H4>
<input TYPE="hidden" NAME="recipient" VALUE="[e-mail]">

</td>
</tr>
<script language='javascript'>
function verifyMe(){
var msg='';

if(document.getElementById('Name').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('Title').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('desisionmaker').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('decisionmakerstitle').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('Company').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('industry').value==''){
msg+='- Text:\n\n';}

var email=document.getElementById('email').value;
if(!(/^\w+([\.-]?\w+)*@\w+([\.-]?\w+)*(\.\w{2,3})+$/.test(email))){
msg+='- Invalid Email Address: '+email+'\n\n';}

if(document.getElementById('email').value==''){
msg+='- E-mail\n\n';}

if(document.getElementById('website').value==''){
msg+='- Website\n\n';}

if(document.getElementById('telephone').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('cellnumber').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('address').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('city').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('country').value==''){
msg+='- Text:\n\n';}

if(document.getElementById('subject').value==''){
msg+='- Text:\n\n';}

etc.

if(document.getElementById('Body').value==''){
msg+='- Text:\n\n';}

if(msg!=''){
alert('The following fields are empty or invalid:\n\n'+msg);
return false
}else{
return true }

}
</script>
<form name='Text' action='/cgi-bin/formmail' method='POST' enctype='multipart/form-data' onsubmit='return verifyMe();'>
<p align="left"><b> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Text:</b></A></p>

<table width='80%' class='table_form_1' id='table_form_1' cellspacing='0' border='1'>
<tr>
<td align='left' class='ftbl_row_1' ><LABEL for='Name' ACCESSKEY='none' ><b><FONT SIZE="1">*</FONT></b><FONT SIZE="1.5">Your Name:</FONT>
</td>
<td align='left' class='ftbl_row_1a' ><input type='text' name='Name' id='Name' size='45' maxlength='45' value=''>
</td>
</tr>

etc.

[tedster]

Tedster, have you employed this method with success?

Yes - 99% reduction in form spam. but I also like Lord Majestic's approach a lot. Maybe combine the two and get a 100% success?

Again, the explanations sound good but more detailed instructions are needed for the HTML/PHP challenged.

Sorry hermosa, I think you're asking for way too much from an HTML discussion forum. It's like asking how to drive a car as part of wanting the directions for getting to Dallas! Where do we start, you know?

You should be able to follow up on the ideas in this thread "on your own" -- or if you really don't want to grow your own knowledge in these areas, then you could hire someone to apply the ideas for you.

[Lord Majestic]

The approach I took was very very successful - the only thing to remember in it is to avoid printing full line of HTML with that hidden variable because bot parsers can (and did in my case) parse it thinking it was HTML and they included it, but if you break that into parts then bots won't get it:

Here is my code:

<script>
<!--
var sN="ns2";

document.writeln('<input '+'type=hidden name='+sN+' value='+'1>');
-->
</script>

So the login/register forms would check to ensure ns2 variable was supplied with value 1 - if not then its 99% a spam bot or 1% legit user without javascript turned on - adding noscript bit should help warn users in advance :)

[piskie]

I find most Spambots include a full http link in the content so my PHP form processor rejects any message containing </a> and this sorts out 99% of those. Now I will combine that with Tedsters css hidden input and hopefully that will sweep up most of the remnants.

[hermosa]

Tedster, I want to grow my knowledge but I am not a techie so I need it broken down in really simple terms or I need to be pointed somewhere that has it broken down step by step.

[timestocome]

On WP I've been blocking all user registrations where the accept statement from the ua = */* That's been blocking about 99% of the problem. Not sure if it will work as well on a forum?

[tedster]

Hermosa, a good place to learn the basics of most online technologies is [w3schools.com...]

[penders]

tedster: Another similar approach uses external css to HIDE an unneeded form input. Anytime that box is filled in, it's a spambot so you just throw the submission out.

Yes, I have used this approach too - very effective. However, I have recently encountered a problem which meant that a legitimate message got flagged as spam! I was using a common field name (name="your_name") to potentially trick the robot to complete the form field. But I think some kind of auto-complete feature of the users browser (Google Toolbar or other plugin perhaps?) was auto filling the field and it was being sent with everything else - without the user knowing!?

Possible solution... use a field name that is a bit more obscure/unique. (Send potential spam messages to an alternative email for later checking if need be, don't simply discard it. Although you might need a massive mailbox - gmail perhaps?)

But the thought occurs that this is a bit of a security issue for the user, if the browser auto completes these 'hidden' (by CSS) fields?! Just speculation, but... a malicious form could have 'hidden' fields for "your_name", "your_address", "your_credit_card"....! Hhhhmmm... if you're browser has such an auto-complete feature it is probably advisable to disable it?! (NB: This isn't the normal auto-complete feature that usually only suggests previous values once you start typing.)

[hermosa]

Do you think that if I change the name of the page on which my form is located it would make any difference? It would be a TON of work to re-code every page linking to it but if it would work, I would try it.

[edited by: hermosa at 1:07 am (utc) on Aug. 14, 2008]

[Lord Majestic]

Do think if I change the name of the page on which my form is located it woud make any difference?

This will involve a lot more work and also less reliable than the other methods offered above.

[hermosa]

Well I guess I have no choice but to hire someone. I created a test form using the resource Tedster created and it doesn't work.

The last part of the code is showing up:

"; } ?>

and I can't even figure out why.

[venti]

We have also used another method involving a minimum delay that the fastest real person could possible fill out the form (i.g. 1.5 seconds), forms submitted quicker than this are bots and ignored.

[penders]

hermosa: The last part of the code is showing up:

Is PHP available on your server? Files containing PHP will generally need to have the extension ".php" so that PHP will parse the files on the server. Your formmail script looks as if it could be written in Perl (action='/cgi-bin/formmail').

venti: We have also used another method involving a minimum delay that the fastest real person could possible fill out the form (i.g. 1.5 seconds), forms submitted quicker than this are bots and ignored.

I like this idea. Presumably you simply store the start time (the time the page is generated) in a type="hidden" field in the form?

[hermosa]

I assumed that php is available. I guess I had better check. I created a PHP file and an HTML file. For some reason some of the code is appearing on the actual HTML page.

"; } ?>

and I can't make it disapper. This is too complicated. Simple HTML I can handle but this is way out of my depth. All of this trouble and now finaincial expense to get someone to code the pages due to spammers.

[hermosa]

Okay, here is what I have done as a temporary fix. I experimented to see if it was the address or the form that was getting spammed. It was definitely the form. So, I created a new page with a different address for my form. Re-coded of my pages to link to it and left the original with a different address. So far it seems to be working. Let's keep our fingers crossed. Anything that comes to that address will be spam so I will just go in once a week and delete it. After a while, if this works and no bonafide inquiries have slipped in, I will just set it to delete everything that comes to that address.

What I will do when I have the budget, is hire someone with a knowledge of PHP to re-code my page and implement some of the suggestions from this thread for a more permanent fix.

I would like to thank everyone for their inpu.

[incrediBILL]

Stopping spam and spambots its about as simple as implementing site wide javascript navigation which is obfuscated and can't be crawled, including obfuscating links in javascript.

When a normal search engine like Google crawls you cloak in the <noscript> area with the plain vanilla navigation and links.

Problem solved, site incapable of being crawled by outside sources such as spam harvesters.

Then you make sure you set your entire site to NOARCHIVE which removed the CACHE pages of the search engine so they can't harvest your site in at the search engine level.

Last, but not least, you can secure your email address by using a submit form and then secure the submit form with a combo of javascript, simple captcha's. When people type in the form I create a "key" in javascript that is sent to the submit page. Spammers that post direct to the form page do not type, they also don't run javascript, therefore they don't create that key so it stops 'em dead.

I have lots of other tricks and filters and tests installed, but that's the basics.

Say bye bye to anyone crawling the site you don't want to crawl it and kiss spambots adios.

[Kufu]

Another similar approach uses external css to HIDE an unneeded form input. Anytime that box is filled in, it's a spambot so you just throw the submission out.

I've tried this with great success.

Also, if you have an email address such as [e-mail], change it to something like [e-mail]. This helps cut down spam by a lot.

[slef]

My main methods are multi-step forms (first page is the form, second is a preview/confirmation) with hidden fields, changing field names, a delay in showing the preview page (slow down spammers) and rate limits (no-one needs to submit 30 contact forms per minute). I get almost no web form spam, but don't seem to have any reduction in real forms.

[Trav]

I'd weight in on the side of a spam-trap:

Another similar approach uses external css to HIDE an unneeded form input. Anytime that box is filled in, it's a spambot so you just throw the submission out. - tedster

You just create a field or two that's hidden via CSS- you can also label these with something like "Don't fill this out, unless you're a robot!"- again, concealed from the standard view by CSS

Anytime a submission comes in with those fields filled in, you can pretty much guarantee a robot completed it-

I have these submissions redirected to a folder that I can periodically review to ensure that i'm not trapping anything important.

Trav

[drkrueger]

Is a form generated, e.g. through PHP echo, vulnerable to being spammed by the bots or is it safe?

Any suggestions on how to prevent mailto: email address from being harvested?

Many thanks,
Doug

[MidwestWebGuy]

You can also check the referring page.

If its not example.com/myform.asp then dont allow the submission. this stopped all the spambots for me.

[Lord Majestic]

Good idea about referers, though one must bear in mind that some security packages remove referers for "security" reasons.

[penders]

Is a form generated, e.g. through PHP echo, vulnerable to being spammed by the bots or is it safe?

An HTML form generated by PHP (or any server-side language) is the same as a static HTML form. Both are subject to the same form of attack (no pun intended). However, using PHP (or any server-side language) to generate the form and consequently to process it's content will enable you to implement some of the methods talked about in this thread.

Any suggestions on how to prevent mailto: email address from being harvested?

Use a form instead. ;) If you still want a mailto link, consider using JavaScript to write your link. Spambots tend not execute JavaScript. However, the email address should at least be human readable if JavaScript is not executed.

[drkrueger]

Penders, if I thought about it for a second longer, I would realize that forms are HTML even if created in PHP. Your point about using PHP to implement some of the defenses is good. So thanks.