homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How to keep dup content from sub domain out of index
francie brady

 6:03 pm on Apr 16, 2010 (gmt 0)

I am faced with this situation:
The asp site has a main domain and a sub domain providing the same content. The sub domain is needed to provide particular affiliates links coming in with a page that contains the appropriate 'buy through this network' information... other than that, the pages are the same as on the main domain.

If we put a disallow on the subdomain robots.txt file could Google not 'find' and index the dup content via external links?

We could put a disallow on the subdomain robots.txt file AND rel canonical tags on each of the sub domain's pages showing the original content as being the main domain's pages - would that be the way to do it?

Thanks for helping :-)



 12:40 am on Apr 17, 2010 (gmt 0)

The robots.txt disallow stops the pages being crawled, so their content isn't fetched. The URLs can still turn up in SERPs as URL-only entries.

Adding the meta robots noindex tag to the pages allows them to be crawled, but nothing about those pages will turn up in SERPs, not even the URL.

The rel="canonical" tag would act as a hint to Google to index the other URL version. They don't have to follow that hint, but usually do.

Use just one method. You cannot combine them.

francie brady

 3:15 pm on Apr 17, 2010 (gmt 0)

Thanks for replying g1smd :-)

It appears, if I dynamically (it's an asp site) "add" a meta no index tag to all the sub domain pages - that should do it?

I sure don't want to lose the indexing of all the main domain's pages, which are identical. However the main domain's pages wouldn't have the dynamically added no index meta tag.

Does this sound ok?

I'm nervous about suggesting an implementation like this unless I'm sure it's correct :-)

Thanks :-)

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved