Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Advanced robots.txt question

About blocking subdomain on different server.

5:47 pm on Feb 14, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 23, 2004
votes: 0

I have a development server at test.domain.com on a separate dedicated server from my live site, www.domain.com.

test.domain.com is a mirror dev site. It's not linked to from anywhere but google has indexed it. For fear of dup content, I don't want Google to crawl it.

How do I exclude the test.domain.com? Do I block it using the robots.txt from the www.domain.com (which is on a different server)? Or do I block it from the robots.txt from the subdomain of the actual test.domain.com server?

5:49 pm on Feb 14, 2005 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
votes: 6

test.domain.com is a separate server to your main domain. So, you must place the robots.txt in the document root of the test server.
7:00 am on Feb 15, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 23, 2004
votes: 0


I do have a few other subdomains on the live server, like region.domain.com.

If I block robots from the dev subdomain, will the crawling of my other live subdomains be affected at all?

Am I correct to assume that blocking the root of the test server only applies to that subdomain?

Disallow: /
So the above, when placed in robots.txt in the root of the test server whose url is test.domain.com, will only apply to that subdomain.

I know I seem overly paranoid, but I don't want to screw anything up.

12:57 am on Feb 21, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 11, 2002
votes: 0

Buddha, you are right.

The directives within a robots.txt file apply exclusively to the (sub)domain where the file resides.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members