homepage Welcome to WebmasterWorld Guest from 54.235.227.60
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Disallowing Robot's Effect on Sub-Domain
Will it disallow the sub domains as well
bppilot




msg:1528509
 4:39 am on Jul 14, 2004 (gmt 0)

I've got a main domain www.example.com that I don't want any spiders to visit (since it's a duplicate of another site I run and don't want the duplicate site penalty - it's just a much easier url for people to remember than my main one).
Now, I've got a few 'sub-domain' sites that I still DO want the spiders to visit, ex:

abc.example.com
def.example.com

Now, if I dissallow robots to www.example.com, will they still index abc.example.com, def.example.com, or will they consider all of those 'dissallowed' since they are under the www.example.com domain?

Hopefully someone out there has this situation in place on there site and knows the outcome so that I don't just need to take the blind leap of faith to find out.

Thanks for any help on this.

BPPilot

 

jdMorgan




msg:1528510
 5:53 am on Jul 14, 2004 (gmt 0)

You just need to be sure that each subdomain returns its own unique robots.txt file, and put the appropriate exclusions in each robots.txt file.

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved