lucy24 - 5:35 pm on Nov 26, 2012 (gmt 0)

Do you want no-crawl or no-index? If no-index, add this simple package to your htaccess or config file, changing the endings as appropriate:

<FilesMatch "\.(js|txt|xml)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

Also good for preventing them from indexing your robots.txt and sitemap, since you obviously can't block them from crawling those ;)