Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo now violating robots.txt

wilderness - 2:53 pm on Dec 29, 2007 (gmt 0)


Has anybody used:

SetEnvIf User-Agent "Slurp/3.0;" keep_out
or

RewriteCond %{HTTP_USER_AGENT} "Slurp/3.0;" [NC]
RewriteRule .* - [F]

and seen any change in either their Yahoo page listings or crawls from the other Yahoo bots?

Thread source:: http://www.webmasterworld.com/search_engine_spiders/3536187.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com