TheMadScientist - 8:28 pm on May 30, 2010 (gmt 0)
That's a good note, and I think I remember reading about that, but for some reason I think it works the same as a 'noindex' on the pages, but what I'm not sure about is if it stops them from crawling the page too... I think it doesn't much, like the X_ header directive you can use rather than the on-page noindex tag.
Please let me know if you've read it stops them from accessing the page too, because it's been a while since I've read anything about either and I think they're '6-of-one, half-dozen-of-the-other' type uses. IOW: I think they all do exactly the same thing, but I could be incorrect on that point.