Master List of Malicious Robots

Is there an active list of the bots we should block?

[Eric_Lander]

I'm looking to find a good resource that keeps an active list of the bad robots up to date. I'm working on a large dynamic site with lots of images and content... And I know if the wrong bots come through, it would cause a lot of harm to the site owners.

Does such a list exist? I used to use an application that was kept up to date, but the creators have since gone out of business.

[DaveAtIFG]

Here's a looooonnnng discussion [webmasterworld.com] that includes a pretty comprehensive list. And our Search Engine Spider Identification [webmasterworld.com] discusses individual crawlers in detail.

[Eric_Lander]

Very cool Dave... Much thanks!

[helohelo]

mmm, i dont understand, this list has to do with httacces, or vcan it also be used for robots.txt

And how malicious can they be? When is a robot on this list

[Dreamquick]

A truely malicious robot will just ignore everything you have in robots.txt so no, you can't stop all malicious robots using robots.txt, you need to use some kind of logic or check to catch them.

What robots.txt does allow you to do is exclude the nuisance/semi-legitimate 'bots that are willing to follow your instructions.

- Tony