Yahoo YSM bot being bad - Please help!

[inbound]

This is a copy of a post in the Overture forum, but I think it's worth putting here too (and getting a different perspective - specifically, should Yahoo be asking for a sustained 8 pages a second?):

Having uploaded 60,000 keywords we've got a little problem.

The automated YSM landing page checker is hitting our little server pretty hard. The pages are all pretty complex and rely on a few external XML feeds, hence a sustained fetching of 8 pages a second is making the system keel over.

How fast have you seen this go? Is our experience a glitch or is it normal for it to go so fast?

I'm guessing that Yahoo maybe are expecting us to have better web-serving resources due to having an innacurate advertiser status of Diamond (The highest status - maybe they think we're bigger than we are).

Has anyone fixed such a problem?

Thanks

[volatilegx]

Yahoo! supports crawl-delay [help.yahoo.com]. You might try that and see if it helps.

[yoyo8]

I'm noticing this, too. Over 100 requests per minute sometimes from 1 Yahoo ip address. This is crazy, no?