I began noticing traffic to my robots.txt page and by looking at Google I can see that they're indexed my robots.txt page. It has no added text and is fairly short yet there it is.
I'm sure this has happened before but what I found worthy of posting here is that the TITLE google has assigned my robots.txt page is the actual content of the file up until reaching the maximum number of characters after which it shows ...
The description is, you guessed it, the contents again. Fubar.
Msg#: 4490461 posted 2:22 am on Sep 2, 2012 (gmt 0)
how to attach a "noindex" directive to something that isn't html
There is a technology called an x-robots-tag that allows a noindex directive to be placed in the http header that's sent by the server. It's very handy for non-html document types, such as video files, pdf files, etc.
Msg#: 4490461 posted 5:58 am on Sep 2, 2012 (gmt 0)
robots.txt page is such an unique name and it should be easy for them to exclude it from their index without any directive. But these days they are so focused on user ex. you know....
The interesting part is what does it rank for to get the traffic? Is it some file or folder name that is unique and which you won't find easily elsewhere on the web or is it a keyword that does drive some traffic to sites?