Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Twiceler/cuil.com craziness (FWIW)


wilderness - 2:21 pm on Dec 7, 2009 (gmt 0)


I've this thing requesting robots.txt and not proceeding any further (it wouldn't get in anyway) for more than a few days now.

This IP's make multiple requests and in no specific order, however quite close together.
216.129.119.zz
67.218.116.zzz


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4038054.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com