BrainDed - 6:00 pm on May 18, 2011 (gmt 0)

My goal is to get google to stop crawling specific URL's and setup a accurate sitemap. I am running a phpbb3 message board with SEF URL's. The problem I have is the forum script generates a URL for every reply in a topic, basically anchors.

This creates 1000's of useless URL's in the eyes of the search engine, even though the users like them for bookmarking.

TOPIC = Domain brewerscubs.com/messageboard/milwaukee-brewers/carlos-gomez-16796.html

Direct Link to post = brewerscubs.com/messageboard/milwaukee-brewers/carlos-gomez-16796.html#p412994

I have been researching and trying to find a way to tell robots.txt to disallow any url containing "#p" but have not had any luck. Also, my host, siteground, is busting my marbles about CPU usage from the testing i have been doing with a gsitecrawler so my days of testing are numbered... I need to get it right this time, so i turn to the experts :)