-- Search Engine Spider and User Agent Identification
---- Thousands of Spambot IPs Hitting my Site
lucy24 - 8:08 pm on Dec 16, 2011 (gmt 0)
it's showing a 404 as I have blocked empty referrals to that page using htaccess. So the bot is not able to access that file.
404 isn't "blocked", it's "can't find". If a malign robot hits a 404 before it gets to the blocking stage, your error logs may show both: the original 404 followed by a 403 meaning that it wasn't allowed to see the 404 page. (No, it does not go into infinite redirect if it's not allowed to see the 403 page. It just goes to the Apache default.) But a custom page doesn't have to be very big. Just, ahem, 513K.