Google indexing robots.txt file

[kunwarbs]

Interesting to see that Google has indexed and cached robots.txt file of reputed websites like nytimes, BBC and Google itself...

[google.com...]

[Brett_Tabke]

ya, mentioned alot in the 4 years they've been doing it.

[NedProf]

Is that because of the text/html mime-type in stead of the text/plain that it should be?

[g1smd]

It's because someone somewhere links to that file, so they treat it as content as well as its true purpose.

[Jordo needs a drink]

It's because someone somewhere links to that file, so they treat it as content as well as its true purpose.

The best example is in the search results you posted. #1 is Wikipedia expaining robots.txt. #2 is the White House robots.txt itself.

Look again at at the Wiki article and you'll see they link to the White House robots.txt

[etgsgroup]

Why Google database show robots.txt file?

[GaryK]

Why Google database show robots.txt file?

That sure seems like an obvious question to me too. It seems like it would be simple enough for Google to implement. Is anyone really interested in seeing the contents of a robots.txt file in their SE results?

[Tastatura]

Number 4 is BT's robots.txt blog :)
number 5 is google's own robots txt file

Webmasterworld: Robots.txt
Brett Tabke experiments with writing a weblog in a text file usually read only by robots. Trenchant commentary on the world of search engine marketing.
www.webmasterworld.com/robots.txt - 2k - Cached - Similar pages

google's robots txt - [ Translate this page ]
User-agent: * Allow: /searchhistory/ Disallow: /news?output=xhtml& Allow: /news?output=xhtml Disallow: /search Disallow: /groups Disallow: /images Disallow: ...
www.google.com/robots.txt - 3k - Cached - Similar pages

[smells so good]

It's one of the few ways that Brett will have his blog found.