homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robots.txt Disallow Multiple Level Subdomains
Can a parent subdomain's robots.txt disallow spiders in a deeper subdomain?

 3:24 pm on Apr 4, 2013 (gmt 0)

If a site uses multiple level subdomains (subdomain2.subdomain1.domain1.com), does each subdomain need to have a robots.txt to disallow the content or can a parent robots.txt file disallow a child?

My understanding is that each subdomain needs to have it's own robots.txt. Are there any exceptions?

I found an old Google Groups post [groups.google.com] where a Google rep said:
When a spider finds a URL, it takes the whole domain name (everything between 'http://' and the next '/'), then sticks a '/robots.txt' on the end of it and looks for that file. If that file exists, then the spider should read it to see where it is allowed to crawl.

If that's correct (& current), it sounds like there must be a robots.txt at subdomain2.subdomain1.domain1.com/robots.txt if you want to disallow all the content in that subdomain.

Is there any technical way that a subdomain could not have a robots.txt visible in its root and still be disallowed via robots.txt?



 8:00 pm on Apr 4, 2013 (gmt 0)

the robot.txt file must be served from the hostname to which its exclusions apply.


 9:02 pm on Apr 4, 2013 (gmt 0)

If your real question is: Do I need to maintain a separate physical robots.txt file for each of my 800 wild-card subdomains? then the answer is no.

The rewrite is a little trickier than when it's happening on the same (sub)domain, but it can still be done.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved