homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

robots.txt and subdomains
how do I block SE's from my subdomains

10+ Year Member

Msg#: 45 posted 1:47 pm on Jun 4, 2002 (gmt 0)

I have many subdomains for my main site in order to help my rankings. However, I have recently read that subdomains with similar content may be considered spam. The content is different in each subdomain, but I do not want to take a chance of having the subdomains labeled as spam and then hurt the entire site. The subdomains do not have enough value to take this chance.

I do not want to simply take down all of the subdomains. Can I use the robots.txt file to prevent the SE's from further viewing the subdomains? If so, do I put a robots.txt file in each of the subdomains?



WebmasterWorld Administrator 10+ Year Member

Msg#: 45 posted 5:51 am on Jun 6, 2002 (gmt 0)

I would put in a robots.txt to keep spiders from accessing the information if it is very similar. However if the content is different than there should be no need as the content will not look like spam. If there is some overlap on content that is fine. If the content is more than 50% redundant between the subdomains and the main domain, than a robots txt might be needed.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved