Welcome to WebmasterWorld Guest from 22.214.171.124
Forum Moderators: open
As for the other robots - I've seen googlebot make requests that fast. If you can dish it up they will take it. But the bot seems smart enough to slow down if you can't serve the requests that quickly.
Ask Jeeves spider came 'round a few sites yesterday for me, but mostly grabbed robots.txt and left. To date, even on larger sites, I haven't seen their bot do this, though I haven't watched a larger scale dynamic site when the bot struck to know that it wouldn't happen in that situation...
Wonder if it means they are testing something new with their bot? Have you contacted them to let them know about the problem?
The one time or two I've done similar, it was with good results...engines can be responsive when it makes their product better.
(feel free to sticky if you like, but that was all I really wanted to know.) :)
And if you do contact them, please let us know what the response is!
OK, whoops! I sticked you some logs- just ignore if you want.
I will contact them, but I do not want to run the risk of getting banned. Is that possible? I conrtacted FAST once about a robot ignoring robots.txt instructions, and they said, "OK, we will put your site into our exclusions list." Noooooo!
I did also find this robot go into a loop looking for something that it kept requesting 5 times a second for at least two minutes.... and the file is SPECIFICALLY EXCLUDED in robots.txt.
Perhaps they do need an e-mail!