statguy - 2:28 pm on May 24, 2011 (gmt 0)
This is what I do in my httpd.conf file :
SetEnvIfNoCase User-Agent "XYZ" bad_bot
SetEnvIfNoCase User-Agent "ABC" bad_bot
Deny from env=bad_bot
So I look at my access log, identify the robots that I want to exclude and add a line in the above, replacing XYZ with the name of the bot in my logs (or a substring of the name that is unique to this robot).
Does that make sense ?