Page is a not externally linkable
jdMorgan - 5:51 pm on Feb 10, 2009 (gmt 0)
Be aware that this only works with the major search engines. It is an undocumented, semi-proprietary "extension" to the original Standard for Robot Exclusion, and you cannot count on it to work with any search engine unless they explicitly say they support it on their "Webmaster Help" page. So if you use this, you must decide on a method --and implement that method-- to handle robots that do not support this Disallow syntax in robots.txt. Another suggestion that we often see is to use the Google, Yahoo, and MSN "Webmaster Tools" to remove unwanted URL listings from the SERPs. Again the problem is, what about all the other search engines? As above, I recommend a 301 redirect to the same URL-path, but with the query string removed, rather than using a non-universal Disallow syntax in robots.txt or any particular search engines' "Webmaster Tools." Jim
Disallow: /*?