lucy24 - 3:04 am on Dec 28, 2011 (gmt 0)
There is no law against it, after all.
Like the man said, an htaccess file beats robots.txt every time.* Maybe Preview thinks of itself as a sort of honorary human.
... and another quick detour tells me that the same applies to Translate. You can feed in the address of a roboted-out file and the translation comes back, large as life. I don't mind that so much, though, because it starts with actual humans requesting the real page. Or does Translate, like Preview, keep cached versions of the most popular sites? Hmmm.
Corollary thought: I am going to need to spend some time studying obscure HTML markup, because I need something that says "Do not translate this part".
And they don't necessarily have "Bot" or "bot" in the UA
... which is why I added "Web\ Preview" ;)
* Or maybe it was "A loaded .45 beats four aces." Something like that, anyway.