Forum Moderators: goodroi

Message Too Old, No Replies

Should I use the "?" or not.

         

kgormat

3:27 am on Dec 12, 2005 (gmt 0)

10+ Year Member



I want to block off all URLs structured as such:
/product-tools.php?tool=big-picture&product=6

Should I use -
Disallow: /product-tools.php

or

Disallow: /product-tools.php?

either, neither, or something else?

jdMorgan

3:52 am on Dec 12, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member




Disallow: /product-tools.php

should work fine.

Spiders interpreting robots.txt use simple prefix-matching; That is, anything that *starts* with "/product-tools.php" will not be fetched.

The only reason you might have to add the "?" is if you wanted spiders to fetch a page called exactly "/product-tools.php" with no query string appended. Even then, I suspect you'd find support for matching the added "?" to be rather sketchy, with some of the second-tier robots getting confused by it.

Jim