Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- spokespider


wilderness - 4:07 am on Aug 13, 2008 (gmt 0)


Pesky bugger.

71.6.45.zzz - - [12/Aug/2008:17:30:51 -0500] "GET /robots.txt HTTP/1.1" 200 4740 "-" "SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really)"

Requested robots and home page six times in succession over four seconds.

Sorry!
Spider (like crawler) is taboo!


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3696941.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com