Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Fake user agent strings


grandma_genie - 5:22 am on Jun 15, 2012 (gmt 0)


Well, now they will be getting a 403.

Only suspicious visitors end up getting that 302 server response. Most typical real visitors come on the site via a Google search and don't hit the index.php file in the root folder. Even the normal search bots like Google or Yahoo or Bing don't access that page. If I go through the logs and look for those GET / HTTP/1.1 or GET / HTTP/1.0 entries, they always come from suspicious IPs and most, if not all, get blocked.

5.9.2.nn belongs to Hetzner Online AG.

Here is another one:
199.168.138.nn - - "GET / HTTP/1.0" 302 - "-" "-"
This is a mail server from VolumeDrive. I'll block them, too.

I could change the coding on the index page to give a 301, but I've just been too lazy to do it. It doesn't happen to Google, Yahoo, or Bing, so for now, since it only happens to the bad guys, I don't think it is an issue.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4462691.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com