lucy24 - 4:39 am on Jun 28, 2013 (gmt 0)
Is it your own server or isn't it? htaccess implies no; "mod_security at server level" implies yes. mod_security seems to add pretty detailed comments to the error log-- but I think they all come through as 500-class errors, so don't spend too much time there.
I took a quick detour to MAMP and tried locking myself out. Even at LogLevel "debug" it still says nothing more than "client denied by server configuration". Grrr.
There are lots of ways to lock people out, but most of them wouldn't apply to a potentially desirable robot. A referer block, for example-- but surely your robots don't come with their own referer?
If there's any chance you blocked them via mod_rewrite, you could try running a RewriteLog and see what turns up. Just to confuse you, there's no on/off setting or LogLevel, you just have to specify a file. But it can't be done in htaccess, so you're looking at restarting your server :(