hi there, more out of curiosity than anything, if we have a site in say 4 languages, www.domain.com/language or language.domain.com, is it possible to write a robots.txt so that a bot will only spider the target language?
for example, have a site in japanese, korean, german and french. i only want google to spider the japanese pages and not the others. how would i write the robot file? can i allow/disallow by using the charset code (assumming they are not all unicode that is) or the html lang = line?