aakk9999 - 11:44 pm on Jun 5, 2013 (gmt 0)
1) Make sure your robots.txt is in plain text format. Having it in UTF-8 will make google not being able to read it and in such case Google acts as robots.txt is not there.
2) Make sure robots.txt returns 200 OK. Returning HTTP 500 on it may result in your site not being crawled and de-indexed.
3) Avoid using parameter "lang" for the language. If you must use parameter, use lng or something else. Omitting & before "lang" parameter makes many browsers and scrappers understanding &lang as left angle bracket <. Even if you correctly use &lang= , scrappers that scrape SERPs scrape it without encoding and then Google picks up duplicate URLs that may look as <=en or similar.
4) Be careful with relative paths - in fact do not use them. Infinite URL space can easily be created with incorrect handling of relative paths, creating thousands and thousands of duplicate pages