Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
bcolflesh - 7:05 pm on May 4, 2005 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/2799.htm
It's been my experience that they try each UA on your site, if they are blocked they switch to a generic Windows + IE UA, so you have to resort to blocking their ranges.
Brought to you by WebmasterWorld: http://www.webmasterworld.com