Robots.txt file indexed in Google SERP

This must be a mistake!

[msr986]

I did a Google search for "robots.txt".

In the middle of the first page, I was surprised to see an actual robots.txt file indexed.

How did something like that get through?

[amoore]

Most people don't include their robots.txt in their robots.txt, so it gets indexed. Unless the search engine makes a special case to not index that file, I imagine that it would be indexed like any other one. I could see google making a special case for the file, but I don't suppose it's really necessary. I suppose that returning someone's robots.txt when you searched for "robots.txt" is a pretty good result.

That being said, there aren't many links to people's robots.txt and there isn't too much content in them, so they probably don't rank very highly for many search terms. Maybe that's why you don't see them too often.

[msr986]

>Most people don't include their robots.txt in their robots.txt, so it gets indexed

I think if this was true, we would see thousands of robots.txt files in the SERP.

I don't have A robots.txt exclusion in any of my robots.txt file, and none of them seem indexed!

[Macguru]

You are right msr986, it is a Googleglitch ©.

But you can find more with allinurl:robots.txt

[ciml]

If people link to foo.txt and it's not excluded, then foo.txt can be listed in Google.

I don't see why /robots.txt shouldn't be listed (other than pandering to the 'security through obscurity' people).

As amoore points out, they don't get into the top 1000 for most phrases.

[pageoneresults]

Hehehe, there will be thousands of people now flocking to optimize their robots.txt file, you watch! ;)

[rfgdxm1]

[google.com...]

Shows as a PR4.

[google.com...]

Google actually *does* have external links to its robots.txt file! Thus, it has honest PR.

[Yidaki]

pageoneresults, nothing new ... i saw one who had more than 6.000 (!) keywords within his robots.txt file - ridiculous!

[Macguru]

And when a visitor hits such files where is he supposed to go from it?

[nutsandbolts]

Well, in the case of the RIAA site that's how it got hacked... the administrator put the admin directory name into the robots.txt file so it wouldn't get crawled - but forgot to password protect it ;)

[dhdweb]

Well, in the case of the RIAA site that's how it got hacked... the administrator put the admin directory name into the robots.txt file so it wouldn't get crawled - but forgot to password protect it

ROFLMAO