Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

How to keep dup content from sub domain out of index

6:03 pm on Apr 16, 2010 (gmt 0)

Junior Member from CA 

10+ Year Member

joined:May 24, 2006
posts: 77
votes: 1

I am faced with this situation:
The asp site has a main domain and a sub domain providing the same content. The sub domain is needed to provide particular affiliates links coming in with a page that contains the appropriate 'buy through this network' information... other than that, the pages are the same as on the main domain.

If we put a disallow on the subdomain robots.txt file could Google not 'find' and index the dup content via external links?

We could put a disallow on the subdomain robots.txt file AND rel canonical tags on each of the sub domain's pages showing the original content as being the main domain's pages - would that be the way to do it?

Thanks for helping :-)
12:40 am on Apr 17, 2010 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
votes: 0

The robots.txt disallow stops the pages being crawled, so their content isn't fetched. The URLs can still turn up in SERPs as URL-only entries.

Adding the meta robots noindex tag to the pages allows them to be crawled, but nothing about those pages will turn up in SERPs, not even the URL.

The rel="canonical" tag would act as a hint to Google to index the other URL version. They don't have to follow that hint, but usually do.

Use just one method. You cannot combine them.
3:15 pm on Apr 17, 2010 (gmt 0)

Junior Member from CA 

10+ Year Member

joined:May 24, 2006
posts: 77
votes: 1

Thanks for replying g1smd :-)

It appears, if I dynamically (it's an asp site) "add" a meta no index tag to all the sub domain pages - that should do it?

I sure don't want to lose the indexing of all the main domain's pages, which are identical. However the main domain's pages wouldn't have the dynamically added no index meta tag.

Does this sound ok?

I'm nervous about suggesting an implementation like this unless I'm sure it's correct :-)

Thanks :-)

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members