Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Why Google Might "Ignore" a robots.txt Disallow Rule


SEOEgghead - 10:48 pm on Aug 15, 2006 (gmt 0)


Out of curiousity, does anyone know if Yahoo and MSN also follow this very literal interpretation of the specification? I would hope so.

I agree that it's wise to put "*" last regardless. I think that cures the ambiguity regardless, right?

I think many programmers would tend to read it the _wrong_ way. I have in the past. It kind of reads like a "switch" statement in C -- where order counts. Arguably, C requires that "*," the default, be last. It's a so-so analogy, but you see my point. If they had used the token "default" instead of "*" nobody would mistake it for a glob/regex like many have ... </rant>


Thread source:: http://www.webmasterworld.com/google/3044757.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com