homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / WebmasterWorld / New To Web Development
Forum Library, Charter, Moderators: brotherhood of lan & mack

New To Web Development Forum

How to prevent search engine indexing
prevent search engine indexing via robots.txt

 12:18 am on Aug 5, 2008 (gmt 0)


I am trying to prevent search engine indexing only on certain pages on a website.

For example, I have www.mysite.com/disclaimer.html, /privacy-policy.html, etc. that I do not want indexed.

How do I accomplish this via robots.txt or any other way?

Thank you.

P.S. I believe I understand how to prevent directories from being indexed as in:
User-agent: googlebot
Disallow: /directory-name

EDIT: the pages are not static html, so I don't think I can use the "noindex" page meta tags

[edited by: Emilio at 12:20 am (utc) on Aug. 5, 2008]



 1:37 am on Aug 5, 2008 (gmt 0)

Is this what you're asking?

User-agent: Googlebot
Disallow: /directory-name
Disallow: /disclaimer.html
Disallow: /privacy-policy.html

Be aware that we have a robots.txt forum [webmasterworld.com] here, for robots.txt-specific discussions.



 6:46 am on Aug 5, 2008 (gmt 0)

register for google webmaster tools to effectively perform these tasks, you need to use robots.txt for adding the above codes..


 12:16 am on Aug 12, 2008 (gmt 0)

For a handful of specific pages, I prefer to use the <meta name="robots" content="noindex,follow"> tag. You can use robots.txt to accomplish the same thing, but I prefer not to clutter it up w/ single pages. But it's a personal preference. Up to you.

Not sure why the page being dynamic prevents you from doing this unless you have no way in your CMS or DB to specify the value for the robots meta tag for each page on your site. We just implemented a CMS system at work, and I made sure that every leaf or branch item type representing a page had a field on it where I can indicate the value. It defaults to "index,follow" but I can select any value from the dropdown.

Global Options:
 top home search open messages active posts  

Home / Forums Index / WebmasterWorld / New To Web Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved