Yahoo to be added to list of bad bots?

Ignoring Crawl Delay - up to 19 pages/sec

[inbound]

Following a suggestion to use Crawl Delay to stop the YSM landing page bot going nuts on my server I implemented the necessary code only to find it ignored.

So the YSM bot has ran through 50,000 (very resource intensive) pages in little bursts. Usually 30 minutes of sustained fetching at 6 pages a second, with the max rate reaching 19 pages a second from a single IP.

Worse still, I got a response from Yahoo stating that there was nothing they could do, they did not see it as an issue!

Any ideas on how to deal with this in the future? (not that we'll be uploading that many phrases at once again)

[volatilegx]

ban the user agent or IP via .htaccess.

[wilderness]

Dan,
He cannot deny access to YSM as he's part of that click through program. (this not the first time he's posted in SSID off-topic).
This should be moved to the Yahoo forum (see his profile).

Don

[inbound]

I'm sorry if people thought I posted off-topic, I thought you would like to hear about a bot that was misbehaving. I clearly stated in the first thread here that it was duplicated in the YSM forum.

It was here that volatilegx suggested a possible solution (crawl delay), which I tried and found out was ignored. It only seemed correct to come back and let you know.

I suppose I may have been hasty in starting a new thread rather than adding it on to the existing one, but I thought a new title would alert more people to the potential problem (as anyone here may also use YSM).

[volatilegx]

Yeah I suppose your results should have been added to the preexisting thread, but its no big deal, really. We'll live ;)