Forum Moderators: open

Message Too Old, No Replies

Special Google Extensions for Robots.txt

Does * and $ work with dynamic URLs?

         

tlhmh1

6:51 am on Oct 1, 2002 (gmt 0)

10+ Year Member



According to: [google.com...]

"Googlebot understands some extensions to the robots.txt standard: Disallow patterns may include * to match any sequence of characters, and patterns may end in $ to indicate that the $ must match the end of a name."

The example given is:

User-agent: Googlebot
Disallow: /*.gif$

I use "?print=yes" on the end of my URLs to bring up the print version of my articles.

Example... "www.widgetworld.com/news/20021001.html?print=yes"

I do not want Google to index my print versions separately and thus see duplicate content. So I thought I might use:

User-agent: Googlebot
Disallow: /*?print=yes$

...in my robots.txt file to prevent Google from indexing them.

Will this work? Will this work with dynamic URLs?