homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

Robots.txt & Sitemaps for multi-country website

Msg#: 4385788 posted 6:23 am on Nov 11, 2011 (gmt 0)

Hi everyone,

I just hope I posted this question in the right category.

If you are doing international SEO for one site i.e. www.website.com for multiple markets and the local sites are adopting a subfolder method, how are you going to view & implement the robots.txt & sitemap.xml files for each of these local sites:

1. www.website.com/fr
2. www.website.com/in
3. www.website.com/sg


Thanks in advance for all the help.




WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 4385788 posted 6:27 pm on Nov 13, 2011 (gmt 0)

Here's how I would approach it. First, be sure to verify each subfolder as "a website" in WebmasterTools.

You only need a single robots.txt served from the domain root. You can get some economies with your Disallow rules by using pattern matching wildcards - in particular the "*" character. If the same URL pattern appears in more than one subfolder, a single rule like this would address all of the prohibited files:

Disallow: /*/[filepath]

For the xml sitemap, you could do the same thing (just use one sitemap at the domain root), but for ease of maintenance, I would probably use a master sitemap.xml at the root that lists only the filepaths for individual sitemaps - one for each of the country-specific subfolders. Then list the location of the master sitemap.xml file in the robots.txt file.

sitemap: http://www.example.com/sitemap.xml

[edited by: tedster at 4:23 pm (utc) on Nov 15, 2011]


Msg#: 4385788 posted 4:15 pm on Nov 15, 2011 (gmt 0)

Got it. Thanks tedster for the step by step guide. Will do it now. Cheers!

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved