Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Twiceler/cuil.com craziness (FWIW)


jdMorgan - 3:09 am on Dec 9, 2009 (gmt 0)


All requests on my sites look normal today -- and that's actually a new thing, because prior to last month, Twiceler apparently did not understand multi-user-agent policy records in robots.txt, and as a result didn't crawl the sites. That's changed now, and they're crawling away (at a normal rate).

Combined with Lord Majestic's speculation above and the "ramp" hosts with no UA, it wouldn't surprise me if they're preparing to roll out a new index some time soon.

Jim


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4038054.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com