Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Disallow: /*? - is it ok ?


Receptional_Andy - 3:16 pm on Oct 9, 2008 (gmt 0)


I've had no issues with that syntax. Note that the safest approach is to only disallow those bots known to understand the wildcard, which I believe is googlebot, msnbot and slurp of the majors.

Unfortunately, the robots standard has developed into not much of a standard at all. I believe the below should work:


User-agent: googlebot
User-agent: slurp
User-agent: msnbot
Disallow: /*?


Thread source:: http://www.webmasterworld.com/robots_txt/3761096.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com