homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Gold Sponsor 2015!
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

excluding dynamic pages in all search engines
excluding dynamic pages in robots.txt

10+ Year Member

Msg#: 413 posted 8:31 pm on Jun 30, 2004 (gmt 0)

I have a site that has one set of html pages and a duplicate set of .php pages. How do I exclude all search engines from indexing my dynamic pages? My site is using .php files that I don't want to have indexed.

Google says to use this:
User-agent: Googlebot
Disallow: /*?

It's my understanding that not all Search Engines use the wildcard. How can I keep my .php files out of the search engines?



10+ Year Member

Msg#: 413 posted 8:58 am on Jul 12, 2004 (gmt 0)


Im also waiting on this question, could the moderator post a response?



WebmasterWorld Senior Member 10+ Year Member

Msg#: 413 posted 2:49 pm on Jul 13, 2004 (gmt 0)

The code you posted will only be recognized/respected by Googlebot, it's non-standard and unrecognized by other spiders. In addition, most other spiders do not recognize wildcards in the "Disallow" since this is also non standard. RE: [searchengineworld.com...]

To insure none of your PHP pages are spidered, you will need to do two things:

1. Move all of them into a unique folder/directory and include the following in robots.txt:
User-agent: *
Disallow: /PHP Folder Name

2. Add <meta name="robots" content="noindex"> in the head section of each page to insure they aren't indexed as a result of spiders following external links to them.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved