homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / WebmasterWorld / Community Building and User Generated Content
Forum Library, Charter, Moderators: rogerd

Community Building and User Generated Content Forum

Preventing unneccessary pages from being indexed

 8:39 pm on Aug 2, 2004 (gmt 0)

This is a rather basic question, but could provide a lot of value to those running their own forums. Search engines will try and index any/all pages possible. On some forums, there are toolbar links within a thread that look like:




Does these forums prevent the search engines from indexing these pages? If so, how is this accomplished?

I looked at the robots.txt file but I didn't find anything there.

[edited by: rogerd at 9:06 pm (utc) on Aug. 2, 2004]
[edit reason] No specifics, please... [/edit]



 10:31 pm on Aug 2, 2004 (gmt 0)

I think it's a good idea to prevent "junk" pages from being indexed - forms, odd versions, etc. Normally these won't be well-ranked, but I'd prefer the SEs to focus on the good content instead of scarfing down low-content pages.

You can restrict indexing with your robots.txt file and/or a robots NOINDEX meta tag. The advantage of the robots.txt approach is that well-behaved bots will never even request the page. The downside is that Google, when it finds the link on a "good" page, may stick that URL in its index for future reference. It won't spider it, but it may keep the entry for it. If you put in the NOINDEX tag, that should prevent if from being listed in any way. Of course, the spider has to request the page to see that tag, so you'll still see some of the bandwidth usage.

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / Community Building and User Generated Content
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved