Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- how to ban Google from indexing ALL .php files


kevsh - 8:44 pm on May 17, 2005 (gmt 0)


Okay, this seems to be along the lines of my issue so I'll post here instead of a new thread.

I have a bunch of incoming links starting with "?" as in:

www.mydomain.com/?=123
www.mydomain.com/?=abc

The query is handled by index.php. Problem is, Google seems to be seeing these as unique pages and not indexing any one of them (possible flagging as duplicate content?)

In any event, I want to block all incoming links beginning with "?" AND any PHP page with a "?" appended, only.

Examples of pages I WANT blocked:

www.mydomain.com/?=123
www.mydomain.com/index.php?id=123
www.mydomain.com/dir/file.php?id=abc

Pages I DO NOT WANT blocked:

www.mydomain.com/index.php
www.mydomain.com/file.html
etc.

I am thinking (hoping) this will work - at least for Google:

User-Agent: googlebot
Disallow: /?
Disallow: /*.php?

If not, any suggestions on how to handle the above scenario?


Thread source:: http://www.webmasterworld.com/robots_txt/622.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com