Welcome to WebmasterWorld Guest from 54.159.190.106

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt for different language sites

can use charset tag to allow/disallow?

   
2:30 am on Jul 7, 2004 (gmt 0)

10+ Year Member



hi there, more out of curiosity than anything, if we have a site in say 4 languages, www.domain.com/language or language.domain.com, is it possible to write a robots.txt so that a bot will only spider the target language?

for example, have a site in japanese, korean, german and french. i only want google to spider the japanese pages and not the others. how would i write the robot file? can i allow/disallow by using the charset code (assumming they are not all unicode that is) or the html lang = line?

thanks

9:31 pm on Jul 20, 2004 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Not quite.

You can disallow listed folders or files, so disallow all except the folder that you want to be indexed.

10:13 pm on Jul 20, 2004 (gmt 0)

10+ Year Member



got it, thanks for the response.
11:51 pm on Jul 20, 2004 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Sorry that it took 2 weeks for someone to reply to your post.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month