homepage Welcome to WebmasterWorld Guest from 184.72.69.79
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt for different language sites
can use charset tag to allow/disallow?
a2ztranslate




msg:1526952
 2:30 am on Jul 7, 2004 (gmt 0)

hi there, more out of curiosity than anything, if we have a site in say 4 languages, www.domain.com/language or language.domain.com, is it possible to write a robots.txt so that a bot will only spider the target language?

for example, have a site in japanese, korean, german and french. i only want google to spider the japanese pages and not the others. how would i write the robot file? can i allow/disallow by using the charset code (assumming they are not all unicode that is) or the html lang = line?

thanks

 

g1smd




msg:1526953
 9:31 pm on Jul 20, 2004 (gmt 0)

Not quite.

You can disallow listed folders or files, so disallow all except the folder that you want to be indexed.

a2ztranslate




msg:1526954
 10:13 pm on Jul 20, 2004 (gmt 0)

got it, thanks for the response.

g1smd




msg:1526955
 11:51 pm on Jul 20, 2004 (gmt 0)

Sorry that it took 2 weeks for someone to reply to your post.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved