Sitemaps, Meta Data, and robots.txt Forum

Most of the time Google does not penalize you (lower the rankings) for duplicate content. They will filter out duplicate content and when that happens they may not keep the url that you prefer. If you want to be sure your preferred url is indexed in Google then block all the other duplicates of it.

For example lets say these three urls are all showing duplicate content:

example.com/post1.html
example.com/post1-printer-friendly.html
example.com/post1-mobile-version.html

You probably would want the cleanest url in Google so you should add the no index tag to print and mobile versions.

It is the duplicate content across other domains that I'm concerned about. For instance, we have a particular TV model where the colorful description is just a copy of what the manufacturer created. So I have that on my site, and using copyspace?, notice a number of other sites used it. I know for a fact we've gotten black listed already.

The good thing is we are really just coming out of the gate still tweaking things and have not added a lot of content yet. So I would hope that we can start correcting the mistakes on just the few things we have, versus having noticed this after adding tons of content. It is still a lot of work if we change the content we do have, that is why I am asking about the No Index No Follow option.

I know the BEST thing to do is fix the content about 200 items.

For my own edification though, when you know flat out that you have a page which has what I now call 'Nuke Dupes'(you heard it here first), where do you put the No Index No Follow?