Forum Moderators: open

Message Too Old, No Replies

What is too much...

         

opulyst

10:50 pm on May 29, 2002 (gmt 0)

10+ Year Member



I run a state soccer association's website and was wondering what was considered too much spidering done by a single spider.

For example in 30 days, the slurp spider has hit their site 134 times. Is this too many? If so or if not what would be considered normal in a month long period like this?

I am using webtrends to track this, so I hope it is fairly accurate.

wilderness

11:55 pm on May 29, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



The slurp spiders are not a good example.
They spider very slowly. Sometimes a page or two at a time. They almost seem to be stuck in a loop. As the spidering travels throughout the entire month.

IMO any bot that travels your enitre site and returns before the period you have "specifed" is too much.
However there are exceptions to the apllication of method.

Too much is a non-standard SE that spiders your entire site in seconds. Or at least a portion before you can stop them ;-)

The SE bots are touchy to mail inquiries and don't seem to speak or understand the same language we do.
I had a problem at one point with the AV Guillver bot and after a few failed communications just gave up communication.
The same with the Lycos Mod spider.