Ask Jeeves hit pretty hard

[carfac]

Got hit (2k+) pretty hard by Ask Jeeves yesterday...

Bad timing- they hit me at about noon... and very excessive... 3-5 hits per second with NO LET UP for a couple hours.

Is this normal for ask jeeves? Is this excessive for a bot?

dave

[fiestagirl]

That seems pretty excessive for the ask jeeves bot. The only time I've ever seen them do that is when they got caught up in a loop (our fault). Most of the time they plod along at about one request every 5 minutes.

As for the other robots - I've seen googlebot make requests that fast. If you can dish it up they will take it. But the bot seems smart enough to slow down if you can't serve the requests that quickly.

[jeremy goodrich]

That is very excessive. Can you give us any more information on the pages? Are they static, or dynamic? And what % of your site did Ask spider?

Ask Jeeves spider came 'round a few sites yesterday for me, but mostly grabbed robots.txt and left. To date, even on larger sites, I haven't seen their bot do this, though I haven't watched a larger scale dynamic site when the bot struck to know that it wouldn't happen in that situation...

[carfac]

Jeremy:

So I do not risk posting any "personal" info, I will sticky you seperately...

I would say they grabbed 5% or so of the site... it is a REALLY big site. Some is dynamic, some static...

dave

[jeremy goodrich]

That's surprising they spidered static stuff that way...hm. Odd.

Wonder if it means they are testing something new with their bot? Have you contacted them to let them know about the problem?

The one time or two I've done similar, it was with good results...engines can be responsive when it makes their product better.

(feel free to sticky if you like, but that was all I really wanted to know.) :)

And if you do contact them, please let us know what the response is!

[carfac]

Jeremy:

OK, whoops! I sticked you some logs- just ignore if you want.

I will contact them, but I do not want to run the risk of getting banned. Is that possible? I conrtacted FAST once about a robot ignoring robots.txt instructions, and they said, "OK, we will put your site into our exclusions list." Noooooo!

I did also find this robot go into a loop looking for something that it kept requesting 5 times a second for at least two minutes.... and the file is SPECIFICALLY EXCLUDED in robots.txt.

Perhaps they do need an e-mail!

dave

[carfac]

Hi:

OK, I send a nicely worded letter on their web feedback form... I guess I gotta hope it gets to someone who cares, and who understands what I meant!

Will update when and if they reply!

dave