phranque - 9:58 am on Sep 16, 2012 (gmt 0)
the <meta> element is part of the Robots Exclusion Protocol.
About the Robots <META> tag:
one disadvantage to using the <meta> tag is it only helps you control indexing of HTML documents, since you can't put such a meta element in a pdf file, for example.
the X-Robots-Tag header is a Google extension that has the advantage of providing the meta robots functionality for web resources that are not HTML documents.
Robots Exclusion Protocol: now with even more flexibility: