Good guy spiders and the Bad ones!

Is there a list

[atlinx]

Obviouslt the are the good visitors that bvisit, but what about the bad ones. The ones that trawl for email addresses etc.

Does anybody have a suggestion for a robots.txt file that will ask the bad guys to go away?

[SlowMove]

robots.txt is for the well behaved.

[bull]

robots.txt will not work for the bad ones.

[webmasterworld.com...]

[cyberkat]

On our domain we sent up a spam abuse education information page with a very extensive list of ISP abuse email addresses for our members to use, and information on tracking down the violator.
Well after that page was up, days later it was crawling with the email harvasters. Funny part is all they do is get abuse addresses and report themselves to all the abuse addresses we list for our members. LOL
As for our guestbook. We do go in and edit email addresses by entering nospam after @.
I found they look for [e-mail] and mailto: in harvesting.

[BillD]

You can find a list of some of the email address harvesters here:
[sendfakemail.com ]
but robots.txt won't do you any good since they will ignore it. You'll need to block them with .htaccess or something similar. Of course, many spammers set their user-agent to look like a regular browser in order to avoid being blocked.

[RealNola]

I setup a formmail.cgi script that does NOTHING.. It just get's logged. Anyone trying that is obviously bad. Block them at the router level OR use your firewall. At what point do you want to start blocking though? The entire class B or just C, maybe just the specific address unless it is an end user (Dialup).

you can set a trap by making the robots.txt file and placing something useless in the file under the disallow pointing to a monster hiding in the closet.

[wilderness]

I setup a formmail.cgi script that does NOTHING

[webmasterworld.com...]