Letting the bots crawl your forums?

Is it a good idea?

[carfac]

Hi:

This relates to search engines and all, but I thought this was a better place to post, as it really is a Webmaster question.

I just noticed a lot of "guest" hits on my forum from 66.77.73.82, which turns out to be Fast. I checked, it is not hitting too fast or anything (twice a minute), but, since their bot does not take a cookie ID, each hit is to a different user.

So, that just got me to thinking.... should I allow SE bots into my forums, or just the main pages....

I know Brett lets spiders here... more than once I have googled a "new" spider from my logs, and the first hit is to WebmasterWorld from Google!

Anyway, I was just curious what you all thought!

dave

[jatar_k]

I would let it spider the whole thing, the more content/pages the merrier. The deeper, more specific info that is more than likely buried in there would see to me to be the best spider food.