to spider or not to spider

should i spider my forums or should i block them

[o0_cops_0o]

On my site i have post forums, just like this one. Should i block it from the google spider, or is a good idea to have the spider index those pages to?

[korkus2000]

Do they have query strings in the urls? If so how many name value pairs are in them?

[o0_cops_0o]

im sorry korkus im new at this and not sure what query stings are. If i dont know should it be better to block the forums?

[chiyo]

Its a good question. We have one site which has a forum attached as a small section. The forum of course produces many pages and even though we have 2,000 other pages of content, its starting to become a larger proportion of the site as a whole. At present we robot.txt the forum pages out, mainly becuase we feel there is a chance that the forum pages may reduce the ranking of our normal pages (i.e. appear above them) which is where we really want people to go. Also, I dont know the risk, but maybe it will reduce the rankings of our other pages in other ways.

The forum is in our cgi folder, and that whole folder is robot txt'd out anyway. So we havent even started to worry about query string complexity etc.

Weve always never really seen any good definition answers on either side but for the moment decided that its best to leave good things alone.

[korkus2000]

querystrings are the part of the url after the?.

www.domain.com/forum/page.asp?name1=value1&name2=value2&name3=value3

That url has 3 name value pairs in its query string. Robots have a hard time with dynamic urls(urls that contain querystrings). If you have many name value pairs in your querystring(like most forums do) then it is harder to get a robot to index them. Google will only spider forums that have querystrings if there is enough Page Rank.The reason I am asking is because many forums on the web with querystrings don't get spidered at all.

I think if you don't moderate your forum frequently then I would not let it be spidered. You never know what could be there. Also like chiyo said, it can also syphone off Page Rank to leave your money pages from getting the Page Rank you want.