Atspider

Spam spider gone mad

[estaquieto]

Saw this in my log:
#reqs: #pages:
1112: 1112: atSpider
1112: 1112: atSpider/1

It hit more than a thousand of my pages looking for email addresses, which reeks of spam! How do I ban a spider from crawling my site (in the robots.txt file)? And how about a complete list of awful spiders that all of us should ban? Thanks!

[PsychoTekk]

malicious bots like those email seeking ones will not care about
the robots.txt, you have to ban them using mod_rewrite
(do a site search [searchengineworld.com], there are plenty of threads about this :))