encyclo - 2:12 pm on Jul 28, 2007 (gmt 0)

From the recent Google Blog posting Robots Exclusion Protocol: now with even more flexibility [googleblog.blogspot.com], Google have announced the availability of the unavailable_after meta element, which enables you to give an expiry date to your pages. (See the thread Google Plans a New Meta Tag - "unavailable_after" [webmasterworld.com] for more information.)

However, there is a second, more interesting, announcement in the same entry: the ability to control Googlebot behavior via HTTP headers rather than on-page meta elements: the X-Robots-Tag header.

We've extended our support for META tags so they can now be associated with any file. Simply add any supported META to a new X-Robots-Tag directive in the HTTP Header used to serve the file.

As mentioned in the post, this is very useful for non-HTML content such as PDF, Word or plain text [webmasterworld.com] files, where you cannot insert meta elements. You can also reduce clutter in the document itself, as well as control indexing via the server configuration rather than editing the files.

One caveat not mentioned by Google is that only Googlebot supports this syntax - unless the other search engines decide to follow suit - so you will still need meta elements for Yahoo or MSN. Also, how long do you reckon we'll have to wait until the first case of a hacked server being modified to send a noindex HTTP header with every request?