lucy24 - 11:00 pm on Jan 14, 2012 (gmt 0)

Part of the answer is simple: As noted in earlier threads, g### does index robots.txt itself.

It is appropriate to allow everyone access to robots.txt. Again, see assorted other threads. But you'd be better with a FilesMatch if you want to group them. I think Apache itself says so. I group mine by extension:

<FilesMatch "(forbidden|goaway|missing)\.html">
Order Allow,Deny
Allow from all
</FilesMatch>

<Files "robots.txt">
Order Allow,Deny
Allow from all
</Files>

<Files "favicon.ico">
Order Allow,Deny
Allow from all
</Files>