Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- Difference between 'Googlebot' and 'googlebot'?


Robert_Charlton - 10:33 pm on May 20, 2009 (gmt 0)


With regard to robots.txt... I'm sure this isn't the last word on this, but w3.org suggests that the question of case sensitivity in the User-agent field is up to the robot. See my emphasis in quote below.

I would assume that, in this area anyway, Google would follow w3 recommendations....

[w3.org...]

Some tips: URI's are case-sensitive, and "/robots.txt" string must be all lower-case. Blank lines are not permitted within a single record in the "robots.txt" file.

There must be exactly one "User-agent" field per record. The robot should be liberal in interpreting this field. A case-insensitive substring match of the name without version information is recommended.

[edited by: Robert_Charlton at 10:36 pm (utc) on May 20, 2009]


Thread source:: http://www.webmasterworld.com/google/3917327.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com