Welcome to WebmasterWorld Guest from

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

robots.txt on a sub-domain

12:09 am on Feb 4, 2010 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 18, 2002
votes: 0

I have a sub-domain to my main domain hosted on a different server to help with load issues. Since creating this sub-domain, it appears that Googlebot is disregarding the robots.txt file from my main site. It's crawling and indexing folders that are specifically excluded.

The robots.txt file I had under my main domain excluded about a dozen folders. The robots.txt file in the root of the sub-domain excluded everything. GWT now shows duplicate content issues because pages were crawled from my excluded folders.

I was wondering if I need the exact same robots.txt file to appear for the sub-domain.
3:19 am on Feb 4, 2010 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 19, 2004
votes: 1

to control the indexing of the sub domain you need to create a bot file on the sub domain not sure if it should be the exact file as i don't know your site.

4:48 am on Feb 4, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
votes: 0

Does your subdomain's robots.txt show up in a browser when you request:


If it does, then you can check the syntax in Google Webmaster Tools to be sure it's saying what you intend it to say. If it doesn't, then you've got the file in the wrong place.