homepage Welcome to WebmasterWorld Guest from 54.237.184.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Robots.txt & Sitemaps for multi-country website
sjcbayona




msg:4385790
 6:23 am on Nov 11, 2011 (gmt 0)

Hi everyone,

I just hope I posted this question in the right category.

If you are doing international SEO for one site i.e. www.website.com for multiple markets and the local sites are adopting a subfolder method, how are you going to view & implement the robots.txt & sitemap.xml files for each of these local sites:

1. www.website.com/fr
2. www.website.com/in
3. www.website.com/sg

?

Thanks in advance for all the help.

Cheers!
Steve

 

tedster




msg:4386487
 6:27 pm on Nov 13, 2011 (gmt 0)

Here's how I would approach it. First, be sure to verify each subfolder as "a website" in WebmasterTools.

You only need a single robots.txt served from the domain root. You can get some economies with your Disallow rules by using pattern matching wildcards - in particular the "*" character. If the same URL pattern appears in more than one subfolder, a single rule like this would address all of the prohibited files:

Disallow: /*/[filepath]

For the xml sitemap, you could do the same thing (just use one sitemap at the domain root), but for ease of maintenance, I would probably use a master sitemap.xml at the root that lists only the filepaths for individual sitemaps - one for each of the country-specific subfolders. Then list the location of the master sitemap.xml file in the robots.txt file.

sitemap: http://www.example.com/sitemap.xml

[edited by: tedster at 4:23 pm (utc) on Nov 15, 2011]

sjcbayona




msg:4387202
 4:15 pm on Nov 15, 2011 (gmt 0)

Got it. Thanks tedster for the step by step guide. Will do it now. Cheers!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved