homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Dissallowing Dynamic Content
Blocking "index.php?"

WebmasterWorld Senior Member trillianjedi us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 3134376 posted 2:14 pm on Oct 25, 2006 (gmt 0)

I'm sure I have asked this before, a couple of years ago, but now can't find the thread ;)

I've been handed a site to fix which has some duplicate content issues. I need to block everything which starts with:-


The "?" is important as I don't want to block:-


I just want to block anything that looks like:-



The key question is can I use the "?" as a wildcard to block all dynamic content in this way?

I suppose a secondary question would be is there an adhered standard by the SE's in this respect? Or should I expect some weirdness from a. n. other engine (which I won't care about too much as long as it works in you-know-which-one).

I note that Google allows the use of an asterisk as a wildcard, although at first glance there appears to be a typo on their page about this which makes me a little nervous:-

To block access to all URLs that include a question mark (?), you could use the following entry:

User-agent: *
Allow: /*?*

From : [google.com...]

Have I mis-interpreted that, or is that a google typo and "allow" should be "disallow"?





5+ Year Member

Msg#: 3134376 posted 6:58 pm on Oct 25, 2006 (gmt 0)

You are absolutely right about the typo. It's been fixed and the examples shown on the help page are now correct. I'm sorry about the confusion.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved