homepage Welcome to WebmasterWorld Guest from 54.237.38.30
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Advanced robots.txt question
About blocking subdomain on different server.
Buddha

10+ Year Member



 
Msg#: 551 posted 5:47 pm on Feb 14, 2005 (gmt 0)

I have a development server at test.domain.com on a separate dedicated server from my live site, www.domain.com.

test.domain.com is a mirror dev site. It's not linked to from anywhere but google has indexed it. For fear of dup content, I don't want Google to crawl it.

How do I exclude the test.domain.com? Do I block it using the robots.txt from the www.domain.com (which is on a different server)? Or do I block it from the robots.txt from the subdomain of the actual test.domain.com server?

 

encyclo

WebmasterWorld Senior Member encyclo us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 551 posted 5:49 pm on Feb 14, 2005 (gmt 0)

test.domain.com is a separate server to your main domain. So, you must place the robots.txt in the document root of the test server.

Buddha

10+ Year Member



 
Msg#: 551 posted 7:00 am on Feb 15, 2005 (gmt 0)

thanks.

I do have a few other subdomains on the live server, like region.domain.com.

If I block robots from the dev subdomain, will the crawling of my other live subdomains be affected at all?

Am I correct to assume that blocking the root of the test server only applies to that subdomain?

Disallow: /
So the above, when placed in robots.txt in the root of the test server whose url is test.domain.com, will only apply to that subdomain.

I know I seem overly paranoid, but I don't want to screw anything up.

LowLevel

10+ Year Member



 
Msg#: 551 posted 12:57 am on Feb 21, 2005 (gmt 0)

Buddha, you are right.

The directives within a robots.txt file apply exclusively to the (sub)domain where the file resides.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved