statguy - 2:28 pm on May 24, 2011 (gmt 0)

Hi,

This is what I do in my httpd.conf file :

SetEnvIfNoCase User-Agent "XYZ" bad_bot
SetEnvIfNoCase User-Agent "ABC" bad_bot
<LocationMatch "/">
Deny from env=bad_bot
</LocationMatch>

So I look at my access log, identify the robots that I want to exclude and add a line in the above, replacing XYZ with the name of the bot in my logs (or a substring of the name that is unique to this robot).

Does that make sense ?