Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Edacious & Intelligent Web Robot, Part 2


abates - 3:17 am on Nov 14, 2006 (gmt 0)


Atleast they've fixed the robots.txt bug I noticed last time, though they're still requesting robots.txt for every page they're downloading (they could at least implement If-Modified-Since for it...)


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3154391.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com