Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- possible botnet

wilderness - 11:47 pm on Jul 11, 2012 (gmt 0)


very unlikely and highly suspicious that a private user would request robots.txt and "nothing else".

68.47.129.zz - - [11/Jul/2012:22:45:09 +0100] "GET /robots.txt HTTP/1.1" 200 2680 "-" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4474787.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com