robots.txt use with vBulletin

[esllou]

trying to get an answer over at vbulletin for two days on this question. Thought I would throw it on the table here and see what you guys make of it:

I want to use robots.txt to keep search engine spiders out of my whole vB forum...except for the archive.

Is this doable...and has anyone done it? I don't even need to allow the spiders onto the main index.php forum homepage, just have links into the archive from my main site.

I want to do this to prevent some of the duplicate content issues that vB currently has with multiple url's leading to same pages (well documented problems)

So, question is...is this a quick and easy fix, are there any potential problems (would only allow Mediapartners in for Adsense reasons) and what would I need to put in my robots.txt?

Many thanks in advance.....

[TheDoctor]

Interesting idea. Apologes for the late reply, but I've been away - busy implementing a vBulletin forum. :)

The URL for the archive board is achive/index.php, whereas the main vBulletin is at index.php. Robots.txt should therefore allow spiders into the subdirectory "archive" but not the main forum directory. Is this possible?

The other possiblity that occurs to me is to put a "noindex" in the "headinclude" template. This is not included when the archive is displayed, meaning that the archive will be indexed but the main forums not.