---- Google is displaying my robots.txt page as a search result
tedster - 2:22 am on Sep 2, 2012 (gmt 0)
how to attach a "noindex" directive to something that isn't html
There is a technology called an x-robots-tag that allows a noindex directive to be placed in the http header that's sent by the server. It's very handy for non-html document types, such as video files, pdf files, etc.