Welcome to WebmasterWorld Guest from 54.196.217.43

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt for different language sites

can use charset tag to allow/disallow?

     
2:30 am on Jul 7, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 24, 2003
posts:99
votes: 0


hi there, more out of curiosity than anything, if we have a site in say 4 languages, www.domain.com/language or language.domain.com, is it possible to write a robots.txt so that a bot will only spider the target language?

for example, have a site in japanese, korean, german and french. i only want google to spider the japanese pages and not the others. how would i write the robot file? can i allow/disallow by using the charset code (assumming they are not all unicode that is) or the html lang = line?

thanks

9:31 pm on July 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Not quite.

You can disallow listed folders or files, so disallow all except the folder that you want to be indexed.

10:13 pm on July 20, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 24, 2003
posts:99
votes: 0


got it, thanks for the response.
11:51 pm on July 20, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Sorry that it took 2 weeks for someone to reply to your post.