lucy24 - 11:23 pm on Dec 11, 2011 (gmt 0)

Can you explain about the "Accept-Language" element? I'm used to seeing it in discussions of redirecting visitors to language-specific pages. What's its role in robot blocking?

Incidentally, I recently added a "Files" exemption for favicon.ico. I did it to make it easier to identify humans in log-processing. (Robots who ask for the favicon do exist, but they're rare, so it alerts me to places where I may have blocked too enthusiastically.) It has the side effect of letting in g###'s faviconbot even though they refuse to put clothes on it :)