lucy24 - 2:45 am on Sep 19, 2012 (gmt 0)

robots.txt is voluntary. .htaccess (or config file) is deaf to all appeals.

The RewriteRule quoted here is one way of exempting robots.txt from rules targeting unwanted visitors. So they can't say "I wanted to obey robots.txt, honest I did, but they wouldn't let me see it!" Another is to put robots.txt into a <Files> envelope to cover anyone blocked by core-level Deny from... directives.

But I don't think it's really necessary in this situation, because how many .txt files have you got? In mod_rewrite, most visitors can be blocked by extension: either .html (or whatever you normally use, including / for directories) or jpe?g|gif|png, depending on whether they are after pages or pictures.

For those just joining us: The ordinary anti-hotlinking routines won't work with pinterest, because their image-harvesting is coded to look as if your own page is the referer.