homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

robots.txt on a sub-domain

 12:09 am on Feb 4, 2010 (gmt 0)

I have a sub-domain to my main domain hosted on a different server to help with load issues. Since creating this sub-domain, it appears that Googlebot is disregarding the robots.txt file from my main site. It's crawling and indexing folders that are specifically excluded.

The robots.txt file I had under my main domain excluded about a dozen folders. The robots.txt file in the root of the sub-domain excluded everything. GWT now shows duplicate content issues because pages were crawled from my excluded folders.

I was wondering if I need the exact same robots.txt file to appear for the sub-domain.



 3:19 am on Feb 4, 2010 (gmt 0)

to control the indexing of the sub domain you need to create a bot file on the sub domain not sure if it should be the exact file as i don't know your site.



 4:48 am on Feb 4, 2010 (gmt 0)

Does your subdomain's robots.txt show up in a browser when you request:


If it does, then you can check the syntax in Google Webmaster Tools to be sure it's saying what you intend it to say. If it doesn't, then you've got the file in the wrong place.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved