Welcome to WebmasterWorld Guest from 54.146.221.231

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Yahoo YSM bot being bad - Please help!

     
4:23 pm on Oct 2, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 9, 2004
posts:1435
votes: 0


This is a copy of a post in the Overture forum, but I think it's worth putting here too (and getting a different perspective - specifically, should Yahoo be asking for a sustained 8 pages a second?):

Having uploaded 60,000 keywords we've got a little problem.

The automated YSM landing page checker is hitting our little server pretty hard. The pages are all pretty complex and rely on a few external XML feeds, hence a sustained fetching of 8 pages a second is making the system keel over.

How fast have you seen this go? Is our experience a glitch or is it normal for it to go so fast?

I'm guessing that Yahoo maybe are expecting us to have better web-serving resources due to having an innacurate advertiser status of Diamond (The highest status - maybe they think we're bigger than we are).

Has anyone fixed such a problem?

Thanks

5:32 pm on Oct 2, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


Yahoo! supports crawl-delay [help.yahoo.com]. You might try that and see if it helps.
8:16 am on Oct 7, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Feb 9, 2003
posts:227
votes: 0


I'm noticing this, too. Over 100 requests per minute sometimes from 1 Yahoo ip address. This is crazy, no?