Slurping me to death..

One ping only, please...

[martyt]

Inktomi's Slurp has been deep crawling my web site all week long - so far I've recorded over 400 hits since Monday and still they keep coming.

Which sounds great until you consider that my entire site has less than 50 pages, all heavily linked to each other. I have query strings on a lot of my URLs, but nothing "dynamic" at all -- the query strings just let me display different products using the same base page.

Slurp has also crawled most pages on my site twice, sometimes 3 times in the last few days.

I appreciate the exposure, but seems like Slurp is a little bit brain-dead at the moment. You'd expect it to be smart enough to recognize the URLs it's already crawled and not hit them again...

Anyone else seeing this behavior?

[lorax]

Yup. Seems to like my outbound news links too. One day I saw it hit the same outbound at least 8 different times with at least 3 different crawlers.