E-mail harvesting

[Determined]

Question: I've got a client who is now blaming their very good search engine positions for an increased number spam messages. Most of the spam is coming in on email addresses that are only published on their site, not addresses that they USE for corespondance, like [e-mail] [e-mail] etc. So how do you stop these rude little email harvesting spiders that ignore your robots.txt? I'm sure I've seen something on it before, but can't remember where.

TIA!

[davek]

I believe the way around this is to use an .htaccess redirect which is done at the server level. Personally I have never used this method but have read some on it and specifically recall this method being used to stop exactly what you are describing, but I will defer to someone who has more knowledge on this method of redirection than I do.. I don't want to give incorrect info.

Dave

[rcjordan]

use this instead of html mailto

 
 <script language=javascript> 
  <!-- 
  var visname = "Site Maintenance"; 
  var recip01 = "tech"; 
  var dom02 = "yourdomain.com"; 
  document.write("<a href=" + "mail" + "to:" + recip01 + "@" + 
                            dom02 + ">" + 
                            visname + "</a>") 
  //--> 
   </script> 

The 'document.write line cannot have linebreaks as it does here.
I haven't come up with a way to do it in forms yet, but using it everywhere else has cut spam to me by 90%

If you have email addresses sprinkled all over the site, you could put this in a js stylesheet and call up email with a function. This would let you change aliases at a central point and have them flow throughout the site... handy if any one email address gets to be a huge problem. Also, in a related note, stay away from "webmaster@" in general. There may be other prefixes ("info@" perhaps) that have a high chance of use so they just a blind spam.

[DaveAtIFG]

Hey Determined

If your hosted on an Apache server this may be exactly what you're looking for...
http://mosa.unity.ncsu.edu/~brabec/antispam.html

[fantomaster]

Here's another option to protect your web side
email addresses from harvesters:

http://fantomaster.com/famshield0.html

Nothing uncrackable, but it certainly does work
wonders for all our sites.

[rogerd]

Let me add one note of caution - most attempts to protect e-mail addresses will have some impact on users. For instance, the javascript method will leave users with JS disabled out of luck (as high as 10 - 20% at some sites). Almost all approaches will leave the user who just wants to copy the e-mail address for later reference without a usable address. So, depending on how important it is for visitors to be able to contact you, it's worth weighing the pluses and minuses of e-mail concealment technology. Personally, I like the approach of plain listing (even using visible text for easy copy/paste by the user) of an special e-mail address used only for web-site replies - by directing this mail to a special mail folder, and performing spam filtering on it, handling the mail is no great burden. On the other hand, if you are publishing many staff mail addresses, then concealment may be worth the risk of occasional loss of visitor contacts.

[EX-S]

This trick is free and easy. Just copy and paste these comments into each page's header. (Or make up some anti spam garbage of your own.)

<!-- WPOISON WPOISON WPOISON WPOISON WPOISON WPOISON WPOISON WPOISON WPOISON --> 
 <!-- mailto:@@@@@@@ MailTo:@.@.@.@ @.@ .@. .@.com .@..com 
 error@invalid@page@exception@hahahaa!!!mailto::  @nospam mailto:@NOSPAM.com [e-mail] 
 [e-mail]  
 spammer at dot dot com  fakeaddress at dot com dot com dot com dot 
 com dot com $?®¶??å?¾ø@hotmail.com &&$@blah.n?t 
 We like it when email harvesters crash   
 -->

How it works:
1. Many spambots are programmed to ignore files with anti-spam software. Wpoison is a well known one.
2. The other garbage is sure to give a spambot indigestion. It might cause it to crash, or give an error message where the user has to click "OK" before the program continues to run. Often these mail harvesters are left running all day and night, so they would lose many hours "work".

Disadvantages
1. The spambots could be programmed to ignore things in comment tags. But that would require extra processing power. The harvester would run much faster just searching for strings that look like email addresses, so either the spammers are slowed, or won't bother looking for comments.

This idea not invented by me, so don't give me the credit for it. It originally came from a "Counter-Exploitation" web site that disappeared somewhere.

Finally, to get around the "need Javascript to display address" problem above, you can use a small graphic depicting your email address in a <NOSCRIPT> section.

Hope that helps!
Jameslak12

[lak12]

I just wanted to add a small note. If you use JavaScript and it spread around - I am sure spammers not gonna sleep on it for long. They'll include this thing in their harvesters and tomorrow it'll stop working. JavaScript is the bad choice for many reasons.
I personally have it off 99% of the time and I won't be able to use your procedure as a visitor to your site.
Second it has no protection against "big guys".
And it's just bad for your site security...
I have spiders written (not for spam reason) that actually kill JavaScripts and harvest valuable information from it as URLs and procedures or forwarding.
Regards.
Mark.

[doulos]

Hate to tell you this but email harvestors are much simplier than you think

One that I'm familiar with simply followsa every link on a webpage building it's "todo list". Then it parses our any string with a "@" symbol in it. Pretty simple

Once you harvest the emails then you run a program that validates each email address removing the garbage. You can open a connection to the SMTP server of each email address to verify the email address.

Plust the email harvesters comes in on port 80 and makes the requests just like any web browser would..

[Xoc]

I wrote about a technique here [webmasterworld.com ].