How to prevent search engine indexing

prevent search engine indexing via robots.txt

[Emilio]

Hello,

I am trying to prevent search engine indexing only on certain pages on a website.

For example, I have www.mysite.com/disclaimer.html, /privacy-policy.html, etc. that I do not want indexed.

How do I accomplish this via robots.txt or any other way?

Thank you.

P.S. I believe I understand how to prevent directories from being indexed as in:
User-agent: googlebot
Disallow: /directory-name

EDIT: the pages are not static html, so I don't think I can use the "noindex" page meta tags

[edited by: Emilio at 12:20 am (utc) on Aug. 5, 2008]

[jdMorgan]

Is this what you're asking?

User-agent: Googlebot
Disallow: /directory-name 
Disallow: /disclaimer.html
Disallow: /privacy-policy.html

Be aware that we have a robots.txt forum [webmasterworld.com] here, for robots.txt-specific discussions.

Jim

[ksumit81]

register for google webmaster tools to effectively perform these tasks, you need to use robots.txt for adding the above codes..

[ZydoSEO]

For a handful of specific pages, I prefer to use the <meta name="robots" content="noindex,follow"> tag. You can use robots.txt to accomplish the same thing, but I prefer not to clutter it up w/ single pages. But it's a personal preference. Up to you.

Not sure why the page being dynamic prevents you from doing this unless you have no way in your CMS or DB to specify the value for the robots meta tag for each page on your site. We just implemented a CMS system at work, and I made sure that every leaf or branch item type representing a page had a field on it where I can indicate the value. It defaults to "index,follow" but I can select any value from the dropdown.