Welcome to WebmasterWorld Guest from 22.214.171.124
It doesn't obey robots.txt in that it ignores the crawl-delay directive.
So far anything I've tried in my .htaccess file doesn't seem to work.
How do you black-hole them? They are showing up in my logs and I just want them blocked.
My hosting is on a linux box, the site is mainly php. So far anything I've tried in my .htaccess file doesn't seem to work.
I am using it without the htaccess file that is included as it was throwing a 500 error and causing it not to work.
In answer to questions above, some of the things I've tried with the htaccess file is redirecting a specific ip to a specific page, blocking proxy sites and most recently blocking access to a specific folder. For the redirect and the folder access I got 500 Internal Server Error, the proxy blocking simply didn't work.
It contains options -indexes, which some hosts already have in place and results in a 500 when added to an htaccess.
Huh?! Do you know this by direct personal experience?