Forum Moderators: goodroi

Message Too Old, No Replies

Multiple "Sitemap: " lines in robots.txt?

Several TLDs use same site code / robots.txt.

         

geoffyp

4:38 pm on Aug 8, 2007 (gmt 0)

10+ Year Member



We have a site where several Top Level Domains point to the same directory/use the same code.
Example:
www.somesite.com
www.somesite.fr
www.somesite.it
www.somesite.es

All go to the same Web directory/pages - but display the site in different languages.

Now, I can build and submit an XML sitemap for each TLD:
www.somesite.com/sitemap.xml
www.somesite.com/FRSitemap.xml
www.somesite.com/ITSitemap.xml
www.somesite.com/ESSitemap.xml

But there is only ONE robots.txt file that is shared between the TLDs.

Can I put the following in the "robots.txt" file?:
Sitemap: [somesite.com...]
Sitemap: [somesite.com...]
Sitemap: [somesite.com...]
Sitemap: [somesite.com...]

Thanks in advance!

goodroi

1:33 pm on Aug 10, 2007 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Adding sitemap location to robots.txt is technically not supported according to the official robots.txt protocol. It is something that the big search engines have been making an exception on. Most of the smaller bots do not support it. Since sitemap support is new and one of the big 3 already had a real hard time handling it already I would not try anything advanced.