Forum Moderators: open
For example in 30 days, the slurp spider has hit their site 134 times. Is this too many? If so or if not what would be considered normal in a month long period like this?
I am using webtrends to track this, so I hope it is fairly accurate.
IMO any bot that travels your enitre site and returns before the period you have "specifed" is too much.
However there are exceptions to the apllication of method.
Too much is a non-standard SE that spiders your entire site in seconds. Or at least a portion before you can stop them ;-)
The SE bots are touchy to mail inquiries and don't seem to speak or understand the same language we do.
I had a problem at one point with the AV Guillver bot and after a few failed communications just gave up communication.
The same with the Lycos Mod spider.