Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- UNTRUSTED in Nokia User Agent

lucy24 - 3:13 am on Dec 18, 2012 (gmt 0)


Auto-referring rises to new heights. Or do I mean plumbs new depths? Two of 'em, about half an hour apart. Guess they were afraid they wouldn't be allowed to see the page if they didn't offer a respectable referer.

182.118.20.222 - - [17/Dec/2012:05:10:32 -0800] "GET /robots.txt HTTP/1.1" 200 657 "http://www.example.com/robots.txt" "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider"

Thread source:: http://www.webmasterworld.com/search_engine_spiders/4440895.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com