Welcome to WebmasterWorld Guest from 220.127.116.11 , register , login , search , subscribe , help , library , PubCon , announcements , recent posts , open posts Pubcon Platinum Sponsor
Advanced robots.txt question About blocking subdomain on different server. Buddha msg:1526249 5:47 pm on Feb 14, 2005 (gmt 0) I have a development server at test.domain.com on a separate dedicated server from my live site, www.domain.com.
test.domain.com is a mirror dev site. It's not linked to from anywhere but google has indexed it. For fear of dup content, I don't want Google to crawl it.
How do I exclude the test.domain.com? Do I block it using the robots.txt from the www.domain.com (which is on a different server)? Or do I block it from the robots.txt from the subdomain of the actual test.domain.com server?
encyclo msg:1526250 5:49 pm on Feb 14, 2005 (gmt 0)
test.domain.com is a separate server to your main domain. So, you must place the robots.txt in the document root of the test server. Buddha msg:1526251 7:00 am on Feb 15, 2005 (gmt 0)
I do have a few other subdomains on the live server, like region.domain.com.
If I block robots from the dev subdomain, will the crawling of my other live subdomains be affected at all?
Am I correct to assume that blocking the root of the test server only applies to that subdomain?
So the above, when placed in robots.txt in the root of the test server whose url is test.domain.com, will only apply to that subdomain.
I know I seem overly paranoid, but I don't want to screw anything up.
LowLevel msg:1526252 12:57 am on Feb 21, 2005 (gmt 0)
Buddha, you are right.
The directives within a robots.txt file apply
exclusively to the (sub)domain where the file resides.