Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Yahoo YSM bot being bad - Please help!



4:23 pm on Oct 2, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

This is a copy of a post in the Overture forum, but I think it's worth putting here too (and getting a different perspective - specifically, should Yahoo be asking for a sustained 8 pages a second?):

Having uploaded 60,000 keywords we've got a little problem.

The automated YSM landing page checker is hitting our little server pretty hard. The pages are all pretty complex and rely on a few external XML feeds, hence a sustained fetching of 8 pages a second is making the system keel over.

How fast have you seen this go? Is our experience a glitch or is it normal for it to go so fast?

I'm guessing that Yahoo maybe are expecting us to have better web-serving resources due to having an innacurate advertiser status of Diamond (The highest status - maybe they think we're bigger than we are).

Has anyone fixed such a problem?



5:32 pm on Oct 2, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Yahoo! supports crawl-delay [help.yahoo.com]. You might try that and see if it helps.


8:16 am on Oct 7, 2006 (gmt 0)

10+ Year Member

I'm noticing this, too. Over 100 requests per minute sometimes from 1 Yahoo ip address. This is crazy, no?

Featured Threads

Hot Threads This Week

Hot Threads This Month