Welcome to WebmasterWorld Guest from

Forum Moderators: keyplyr & mack

Message Too Old, No Replies

How to prevent search engine indexing

prevent search engine indexing via robots.txt

12:18 am on Aug 5, 2008 (gmt 0)

Junior Member

10+ Year Member

joined:May 18, 2005
votes: 0


I am trying to prevent search engine indexing only on certain pages on a website.

For example, I have www.mysite.com/disclaimer.html, /privacy-policy.html, etc. that I do not want indexed.

How do I accomplish this via robots.txt or any other way?

Thank you.

P.S. I believe I understand how to prevent directories from being indexed as in:
User-agent: googlebot
Disallow: /directory-name

EDIT: the pages are not static html, so I don't think I can use the "noindex" page meta tags

[edited by: Emilio at 12:20 am (utc) on Aug. 5, 2008]

1:37 am on Aug 5, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0

Is this what you're asking?

User-agent: Googlebot
Disallow: /directory-name
Disallow: /disclaimer.html
Disallow: /privacy-policy.html

Be aware that we have a robots.txt forum [webmasterworld.com] here, for robots.txt-specific discussions.


6:46 am on Aug 5, 2008 (gmt 0)

New User

5+ Year Member

joined:Aug 5, 2008
posts: 20
votes: 0

register for google webmaster tools to effectively perform these tasks, you need to use robots.txt for adding the above codes..
12:16 am on Aug 12, 2008 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 5+ Year Member

joined:Nov 11, 2007
votes: 3

For a handful of specific pages, I prefer to use the <meta name="robots" content="noindex,follow"> tag. You can use robots.txt to accomplish the same thing, but I prefer not to clutter it up w/ single pages. But it's a personal preference. Up to you.

Not sure why the page being dynamic prevents you from doing this unless you have no way in your CMS or DB to specify the value for the robots meta tag for each page on your site. We just implemented a CMS system at work, and I made sure that every leaf or branch item type representing a page had a field on it where I can indicate the value. It defaults to "index,follow" but I can select any value from the dropdown.