Welcome to WebmasterWorld Guest from 54.205.96.97

Forum Moderators: werty

Yahoo Product Submit Crawlers Not Behaving?

   
11:40 am on May 6, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Anyone out there use Yahoo Product Submit notice that the spiders have two main issues:

(1) The spiders don't seem to be properly identifying themselves
(2) The spiders often crawl a single site at a rate of 20 pages per second (which is incredibly high crawl rate).

Anyone else notice that?

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month