Welcome to WebmasterWorld Guest from 54.146.239.96

Forum Moderators: goodroi

Message Too Old, No Replies

Advanced robots.txt question

About blocking subdomain on different server.

     

Buddha

5:47 pm on Feb 14, 2005 (gmt 0)

10+ Year Member



I have a development server at test.domain.com on a separate dedicated server from my live site, www.domain.com.

test.domain.com is a mirror dev site. It's not linked to from anywhere but google has indexed it. For fear of dup content, I don't want Google to crawl it.

How do I exclude the test.domain.com? Do I block it using the robots.txt from the www.domain.com (which is on a different server)? Or do I block it from the robots.txt from the subdomain of the actual test.domain.com server?

encyclo

5:49 pm on Feb 14, 2005 (gmt 0)

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member



test.domain.com is a separate server to your main domain. So, you must place the robots.txt in the document root of the test server.

Buddha

7:00 am on Feb 15, 2005 (gmt 0)

10+ Year Member



thanks.

I do have a few other subdomains on the live server, like region.domain.com.

If I block robots from the dev subdomain, will the crawling of my other live subdomains be affected at all?

Am I correct to assume that blocking the root of the test server only applies to that subdomain?

Disallow: /
So the above, when placed in robots.txt in the root of the test server whose url is test.domain.com, will only apply to that subdomain.

I know I seem overly paranoid, but I don't want to screw anything up.

LowLevel

12:57 am on Feb 21, 2005 (gmt 0)

10+ Year Member



Buddha, you are right.

The directives within a robots.txt file apply exclusively to the (sub)domain where the file resides.