I searched for pages from my website by the comand site:www.example.com and google listed www.example.com/robots.txt
Is the same for you? Or do I have some problems?
g1smd
5:32 pm on Oct 2, 2011 (gmt 0)
It happens sometimes.
Disallow: /robots.txt
will get it out of the index, without stopping it being read for its original purpose.
lucy24
7:30 pm on Oct 2, 2011 (gmt 0)
Also sitemaps. It has come up in a few threads in recent months. Google's stated position is "If it is an URL, we will crawl it". (Really. It's somewhere in their GWT forums. In fact they seemed offended by the idea that someone might not want them to.) It must make the googlebot insane to know you have an .htaccess -- but they can't crawl that.
levo
7:48 pm on Oct 2, 2011 (gmt 0)
<FilesMatch "\.(txt|xml)$"> Header set X-Robots-Tag "noindex, follow" </FilesMatch>