Welcome to WebmasterWorld Guest from

Message Too Old, No Replies

robots.txt on a sub-domain



12:09 am on Feb 4, 2010 (gmt 0)

10+ Year Member

I have a sub-domain to my main domain hosted on a different server to help with load issues. Since creating this sub-domain, it appears that Googlebot is disregarding the robots.txt file from my main site. It's crawling and indexing folders that are specifically excluded.

The robots.txt file I had under my main domain excluded about a dozen folders. The robots.txt file in the root of the sub-domain excluded everything. GWT now shows duplicate content issues because pages were crawled from my excluded folders.

I was wondering if I need the exact same robots.txt file to appear for the sub-domain.


3:19 am on Feb 4, 2010 (gmt 0)

10+ Year Member

to control the indexing of the sub domain you need to create a bot file on the sub domain not sure if it should be the exact file as i don't know your site.



4:48 am on Feb 4, 2010 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Does your subdomain's robots.txt show up in a browser when you request:


If it does, then you can check the syntax in Google Webmaster Tools to be sure it's saying what you intend it to say. If it doesn't, then you've got the file in the wrong place.

Featured Threads

Hot Threads This Week

Hot Threads This Month