lucy24 - 12:20 am on Jun 6, 2013 (gmt 0)
Make sure your robots.txt is in plain text format. Having it in UTF-8
Format and file encoding have nothing to do with each other. Odd to see this here. It's a pervasive error on my e-books forum.
So long as none of your filenames or directories use non-ASCII characters, the encoding is immaterial in any cases.
robots.txt can return either 200 or 404 (meaning you haven't got one). Anything else, and the well-behaved robot will go away sulking.
Avoid using parameter "lang" for the language.
To be clear: you're talking about URL parameters, right? Not <lang="something"> declarations. I know this one well; it plays havoc with my log-wrangling in exactly the way you describe. Another parameter to avoid is "ni". Can't remember who uses it, or what for-- only that it turns into a mess.