Page is a not externally linkable
Receptional_Andy - 3:16 pm on Oct 9, 2008 (gmt 0)
Unfortunately, the robots standard has developed into not much of a standard at all. I believe the below should work:
I've had no issues with that syntax. Note that the safest approach is to only disallow those bots known to understand the wildcard, which I believe is googlebot, msnbot and slurp of the majors.
User-agent: googlebot
User-agent: slurp
User-agent: msnbot
Disallow: /*?