Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- abot/0.1

bcolflesh - 7:05 pm on May 4, 2005 (gmt 0)


It's been my experience that they try each UA on your site, if they are blocked they switch to a generic Windows + IE UA, so you have to resort to blocking their ranges.

Thread source:: http://www.webmasterworld.com/search_engine_spiders/2799.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com