Welcome to WebmasterWorld Guest from 54.163.25.166

Message Too Old, No Replies

robots.txt on a sub-domain

     
12:09 am on Feb 4, 2010 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 18, 2002
posts:143
votes: 0


I have a sub-domain to my main domain hosted on a different server to help with load issues. Since creating this sub-domain, it appears that Googlebot is disregarding the robots.txt file from my main site. It's crawling and indexing folders that are specifically excluded.

The robots.txt file I had under my main domain excluded about a dozen folders. The robots.txt file in the root of the sub-domain excluded everything. GWT now shows duplicate content issues because pages were crawled from my excluded folders.

I was wondering if I need the exact same robots.txt file to appear for the sub-domain.
3:19 am on Feb 4, 2010 (gmt 0)

Preferred Member

10+ Year Member

joined:Oct 19, 2004
posts:351
votes: 0


Hi,
to control the indexing of the sub domain you need to create a bot file on the sub domain not sure if it should be the exact file as i don't know your site.

Vimes.
4:48 am on Feb 4, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


Does your subdomain's robots.txt show up in a browser when you request:

subdomain.example.com/robots.txt

If it does, then you can check the syntax in Google Webmaster Tools to be sure it's saying what you intend it to say. If it doesn't, then you've got the file in the wrong place.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members