dvduval - 1:52 pm on Jul 29, 2003 (gmt 0)
I have a site that's running a phpbb forum, and google seems to index specific number of pages from the site no matter what I do. The problem is that many of the pages google was indexing were worthless such as the login for private messages. Following the advice of this post...
...I created my own robots.txt file:
It worked like a champ and now Google is getting at the good stuff and leaving the spider poop alone.
Thanks for the tip!