Welcome to WebmasterWorld Guest from

Forum Moderators: brotherhood of lan & mack

Message Too Old, No Replies

How to prevent search engine indexing

prevent search engine indexing via robots.txt



12:18 am on Aug 5, 2008 (gmt 0)

10+ Year Member


I am trying to prevent search engine indexing only on certain pages on a website.

For example, I have www.mysite.com/disclaimer.html, /privacy-policy.html, etc. that I do not want indexed.

How do I accomplish this via robots.txt or any other way?

Thank you.

P.S. I believe I understand how to prevent directories from being indexed as in:
User-agent: googlebot
Disallow: /directory-name

EDIT: the pages are not static html, so I don't think I can use the "noindex" page meta tags

[edited by: Emilio at 12:20 am (utc) on Aug. 5, 2008]


1:37 am on Aug 5, 2008 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Is this what you're asking?

User-agent: Googlebot
Disallow: /directory-name
Disallow: /disclaimer.html
Disallow: /privacy-policy.html

Be aware that we have a robots.txt forum [webmasterworld.com] here, for robots.txt-specific discussions.



6:46 am on Aug 5, 2008 (gmt 0)

5+ Year Member

register for google webmaster tools to effectively perform these tasks, you need to use robots.txt for adding the above codes..


12:16 am on Aug 12, 2008 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member

For a handful of specific pages, I prefer to use the <meta name="robots" content="noindex,follow"> tag. You can use robots.txt to accomplish the same thing, but I prefer not to clutter it up w/ single pages. But it's a personal preference. Up to you.

Not sure why the page being dynamic prevents you from doing this unless you have no way in your CMS or DB to specify the value for the robots meta tag for each page on your site. We just implemented a CMS system at work, and I made sure that every leaf or branch item type representing a page had a field on it where I can indicate the value. It defaults to "index,follow" but I can select any value from the dropdown.


Featured Threads

Hot Threads This Week

Hot Threads This Month