Welcome to WebmasterWorld Guest from

Forum Moderators: werty

Message Too Old, No Replies

Yahoo Product Submit Crawlers Not Behaving?



11:40 am on May 6, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Anyone out there use Yahoo Product Submit notice that the spiders have two main issues:

(1) The spiders don't seem to be properly identifying themselves
(2) The spiders often crawl a single site at a rate of 20 pages per second (which is incredibly high crawl rate).

Anyone else notice that?


Featured Threads

Hot Threads This Week

Hot Threads This Month