Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo! Crawlers - A response from Yahoo! Search
incrediBILL - 8:33 am on Jul 18, 2006 (gmt 0)Thread source:: http://www.webmasterworld.com/search_engine_spiders/3006509.htm
Mokita, this is why you need a dynamic robots.txt only served up by user agent so if they see all bots are blocked and continue anyway, which forgoes any UA mixup, you know they are messed up for sure and can drop them in .htaccess.
Brought to you by WebmasterWorld: http://www.webmasterworld.com