dstiles - 8:01 pm on May 31, 2010 (gmt 0)

An alternative to robots.txt and noindex in preventing a page being read by a bot AND only serving up a very short bit of code is, of course, to return a 404, 403 or 405, according to relevance. Obviously this is no good if the page has "follow" links on it.

If noindex links are commonly found by a bot on indexed pages, leading to a rejection of some sort, then those links could be omitted from the page served to bots, saving their time and your resources.

Of course, if google decides you are cloaking the page then you may have a problem. On the other hand, who are they, really, to say what you can and cannot serve up to visitors? It's YOUR site, not theirs.

I do all of the above on a small scale - none of my sites is very big, anyway - and so far it all works.