Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: rogerd
Does these forums prevent the search engines from indexing these pages? If so, how is this accomplished?
I looked at the robots.txt file but I didn't find anything there.
[edited by: rogerd at 9:06 pm (utc) on Aug. 2, 2004]
[edit reason] No specifics, please... [/edit]
You can restrict indexing with your robots.txt file and/or a robots NOINDEX meta tag. The advantage of the robots.txt approach is that well-behaved bots will never even request the page. The downside is that Google, when it finds the link on a "good" page, may stick that URL in its index for future reference. It won't spider it, but it may keep the entry for it. If you put in the NOINDEX tag, that should prevent if from being listed in any way. Of course, the spider has to request the page to see that tag, so you'll still see some of the bandwidth usage.