---- Difference between 'Googlebot' and 'googlebot'?
Robert_Charlton - 10:33 pm on May 20, 2009 (gmt 0)
With regard to robots.txt... I'm sure this isn't the last word on this, but w3.org suggests that the question of case sensitivity in the User-agent field is up to the robot. See my emphasis in quote below.
I would assume that, in this area anyway, Google would follow w3 recommendations....
Some tips: URI's are case-sensitive, and "/robots.txt" string must be all lower-case. Blank lines are not permitted within a single record in the "robots.txt" file.
There must be exactly one "User-agent" field per record. The robot should be liberal in interpreting this field. A case-insensitive substring match of the name without version information is recommended.
[edited by: Robert_Charlton at 10:36 pm (utc) on May 20, 2009]