Jasp - 12:42 am on May 18, 2008 (gmt 0)

If you use robots.txt to block it, that would mean that googlebot should not even request robots.txt - an insane loop. And of course, you don't use meta tags in a robots.txt file.

We already have a very common situation where the contents of robots.txt doesn't prevent robots.txt actually being fetched:


User-agent: *
Disallow: /


Clearly robots.txt is included under the root directory, but still must be requested and requested repeatedly to see if it has changed.

So just as

Disallow: /*.txt$

would prevent all .txt files from being indexed then so should

Disallow: /robots.txt

prevent that particular .txt file from being indexed without problems.