- - Search Engines
- -- Search Engine Spider and User Agent Identification
- ---- Yahoo now violating robots.txt
wilderness - 4:40 pm on Jan 12, 2008 (gmt 0)
I've added Slurp/3.0 to denies and will thus see how it effects my listings and other crawls.
Just a follow up on this.
This denial has NOT hindered the adding of some recent additions (new pages in the past ten days) of my websites to Yahoo listings.
Thread source: http://www.webmasterworld.com/search_engine_spiders/3536187.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com