tedster - 4:52 am on Sep 28, 2011 (gmt 0)
I would prefer to have these directories redirect to the English pages like they already do. This offers the best user experience, but of course is in conflict with g as I am surely getting pinged for dup content.
The whole robots.txt issue should just evaporate if the redirect is technically sound. The URL should be changing in the browser bar when you request the old URL, and a header checker should show a 301 status for the original URLs.
If the bot gets that experience, then you don't have duplicate content.