Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Yahoo now violating robots.txt
wilderness - 1:14 am on Dec 31, 2007 (gmt 0)
keyplr,
I've added Slurp/3.0 to denies and will thus see how it effects my listings and other crawls.
Thread source:: http://www.webmasterworld.com/search_engine_spiders/3536187.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com