homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Yahoo YSM bot being bad - Please help!

 4:23 pm on Oct 2, 2006 (gmt 0)

This is a copy of a post in the Overture forum, but I think it's worth putting here too (and getting a different perspective - specifically, should Yahoo be asking for a sustained 8 pages a second?):

Having uploaded 60,000 keywords we've got a little problem.

The automated YSM landing page checker is hitting our little server pretty hard. The pages are all pretty complex and rely on a few external XML feeds, hence a sustained fetching of 8 pages a second is making the system keel over.

How fast have you seen this go? Is our experience a glitch or is it normal for it to go so fast?

I'm guessing that Yahoo maybe are expecting us to have better web-serving resources due to having an innacurate advertiser status of Diamond (The highest status - maybe they think we're bigger than we are).

Has anyone fixed such a problem?




 5:32 pm on Oct 2, 2006 (gmt 0)

Yahoo! supports crawl-delay [help.yahoo.com]. You might try that and see if it helps.


 8:16 am on Oct 7, 2006 (gmt 0)

I'm noticing this, too. Over 100 requests per minute sometimes from 1 Yahoo ip address. This is crazy, no?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved