Tell Google about duplicates with NOINDEX?

[Decius]

I have an ecommerce site that has a product, say widgets... and for each widget there is a color then there is a size.

Well, when you update the color, there are seperate sizes for each color, so a new list of sizes shows up.

Now, I know I can do AJAX but I don't want to do that at the current time for various reasons.

It ocurred to me that perhaps Google is penalizing me because of this seemingly duplicate content (because the page with the color and size chosen is almost identical, save the url looks different and there is some info on the particular color and size).

So, can I solve this by telling google NOT to index any of the URLs that include a color and/or size specification?

If so, and people link to me from those pages, will that link help me out on the base product page or will it be lost as a result of the noindex tag?

Thanks!

[g1smd]

Yes, use noindex to keep the duplicates out of the index.

[Decius]

Super!

Now, the more important question which I have yet been unable to find an answer for is this:

If someone links to a page that has "noindex,follow" does the strength of that inbound link pass over to the links on that specific page?

Please answer with some level of proof/experience... not theory.

[g1smd]

As long as the page does NOT have nofollow on IT, then it probably does pass through.

However, I don't think you should be worrying about that minutae so much. It is a minor detail I think.

[Decius]

Well, it's not a minor detail in regards to backlinking. A lot of users will most likely link to these duplicate pages and as a result of that I want to ensure that I'm not wasting all of those links by not indexing it. If those links are wasted, I might find an alternate method of keeping Google out of there.

Regardless, the question still stands. Is PR and backlink strength passed over pages that have "noindex, follow" on them?

Thanks.

[g1smd]

The point is that people will link to the URL that they want to link to. You can't stop that from happening.

Additionally, Google knows that that will happen and more than likely will already have factored it into the system.

In fact, it might be that if your links are "too perfect", that will flag your site up in some way.

[Decius]

Thanks for the responses, but unless you have some more information than your opinion on what Google is doing, please try to get more evidence before responding. I have come to the same conclusion you have, but without more feedback and more evidence I'm not inclined to take it as the most probable solution yet.

So perhaps we should wait for others to respond to this so we get a statistic of what people have experienced.

Thanks.

[trinorthlighting]

All the tag does (Follow or No Follow) tells the search engine weather to follow the links on the page or not to.

As far as PR is concerned, only google knows that answer.

If you are real worried about the PR, then take the time, clean up the duplicate content and 301 redirects.

[theBear]

Decius,

Tell me if PR flows through links then in order to "pass" PR would not one have to follow the links to create the mesh?

nofollow if followed (bad pun) would prevent that, so I'd expect that follow is what you want in order to "pass" PR.

But heck ask Matt on his blog.

[edited by: theBear at 11:09 pm (utc) on Nov. 28, 2006]

[Decius]

The duplicate content is not bad, actually. It is not a mistake or spam... it is, as I clearly, clearly stated above, an efficient method of personalizing a product.

I will try asking Matt, thanks.

[Decius]

I still cannot find a reliable answer on this. Does anyone have any further information in this regard?

[tedster]

Let's summarize the thread so far.

1. Initial question: can a robots noindex meta tag help resolve a duplicate content siutation?
This is answered - yes, most definitely.

2. Follow-up question: If someone links to a page that has "noindex,follow" does the strength of that inbound link pass over to the links on that specific page?
This is much trickier to answer with anything like certainty. One of the challenges is that even though Google behind the scenes uses a PR calculation figured to many decimal places, all that is publicly available is the standard 1 throuh 10. I took a close look at one site where we made extensive use of "noindex,follow" and there is no obvious pattern of PR fall-off on the target pages of links on those noindex pages. But as I said, PR data is not nearly granular enough to be 100% sure.

Backlink influence (anchor text, etc) is even harder to measure with certainty. Google does not report all backlinked linked pages by a large measure, so we are in the dark here as well.

3. Topics not yet brought up: 301 redirect
301 redirect of the near duplicate urls to one chosen url. This definitely can pass any PR and backlink influence that is available. However, it would also redirect user's who click on those backlinks, and this may be a detriment. Solving issue with Google in a way that hurts the end user is not a happy trade-off. Still, depending on what content would be on the final landing page, perhaps the 301 redirect would be acceptable. Only you can decide that - but it is another approach.

4. Topics not yet brought up: robots.txt
This is a bit more drastic, because Google will request a "noindex,follow" page and know bout the links there. With a robots.txt exculsion rule, those urls will most likely be excluded from calculations. Still, it is a strong approach to eliminating duplicate troubles with Google. Depending on how many bklinks are involved, it may be a more foolproof approach.

5. Is there really a "penalty" here?
In my experience, low-level duplicate content such as you've described does not cause a penalty. Instead, the Google algo applies a filter so that only one url can show in the SERP. This is not a penalty, it is a filter. By using a "noindex,follow" meta tag, you are in effect just giving Google your determination of which url should not be filtered out.

So I would suggest you take another look at your situation with Google - perhaps their filtering is already doing well by you, and trying to overly control the situation just might create problems and hurt your traffic. For example, you might make a global decision to noindex one type of url, but in a few cases these are the urls that Google does use to send you traffic. I have seen (and been asked to fix) websites that have dome something like this.

[Decius]

I took a close look at one site where we made extensive use of "noindex,follow" and there is no obvious pattern of PR fall-off on the target pages of links on those noindex pages.

This is quite relevent and I am interested in more personal experiences such as this.

[natural number]

Perhaps you need to add NOINDEX to the duplicate content. One time in band camp, I had the same products (doh!) on two websites. On one site I added NOINDEX, and the rankings improved. The rankings improved on the first site I had to publish the material.

Now as for the tiny variation of color and size, I suppose each page is different then, by only a few characters. This could set off a duplicate content filter. Figure out what the most popular products are and keep them in the index. NOINDEX the least popular products.

However, you could keep everything live if you can add value to each product listing.

take that as my opinion,