Indexing With and Without www. in Url Causing Duplicate Content

[jonescd]

Webmasters,

My site is 5 months old. Yahoo has me indexed with pages with www in the url and with urls that do not have www. What is the best solution to remedy this problem. I am afraid this is causing duplicate content and affecting page rank. Do I have to do an htaccess redirect or should I contact Yahoo to clear the non www. I have yet to submit my sitemap.

Thanks,
SolarChris

[dublinmike]

Stick this in at the top of your .htaccess file:

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST}!^www.yourdomain.com
RewriteRule ^(.*)$ [yourdomain.com...] [R=301,L]

The above pattern matches when the domain requested does not begin with www.yourdomain.com, i.e. if someone tries to access yourdomain.com without the www. The [R=301,L] are important because you'll probably want it to be a permanent redirect and you'll probably want to restart the pattern matching loop (in case you have other rewrites going on underneath which pertain to www.yourdomain.com)...

[jdMorgan]

Small formatting correction and optimization tweak:

Options +FollowSymLinks
RewriteEngine on
RewriteCond %{HTTP_HOST} !^www\.example\.com
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

Jim

[edited by: jdMorgan at 2:30 pm (utc) on Dec. 6, 2006]

[jonescd]

Even with the redirect you gave me, won't there still be a duplicate content problem between the nonwww pages and the www pages having the same content. Will exculding nonwww pages on the sitemap repair this problem

[jdMorgan]

If you 301 redirect from the wrong URL to the right URL, then the "content" at the wrong URL simply ceases to exist in the client's eyes. This is the correct solution as provided for by the HTTP protocol [w3.org], independent of any point solutions offered by search providers.

Jim

[jonescd]

When you say client, does this mean that Yahoo or even Google will not penalize you for duplicate content as long as you do go with the 301 redirect script that was given?

[jdMorgan]

A client is anything that makes request from from your server.

When you do a 301 redirect, the redirected-from URL is seen as obsolete, and "content" at that URL ceases to exist. Instead, requestors of the old URL are told to go get the content from the new URL. Therefore, there can be no duplicate content once the 301 redirect is installed.

The document I linked to above explains this in great detail.

BTW, it's not a penalty, it is simply a splitting of ranking between the two URLs. Since you inform the client that there is only one URL by using the redirect, the problem simply goes away. Penalties are reserved for intentional attempts to get the same content listed in search results at dozens or hundreds of URLs.

Developers of new sites are wise to install a domain canonicalization redirect such as this before putting any content onto the server, thereby avoiding the rank-splitting problem from the very start.

Jim

[jonescd]

jd,

Ok, I have uploaded the htaccess file with the new script to 301 redirect the non-www pages.

Now, I still have a problem with my SSL pages (https) being indexed more than my regular http pages.

What script is necessary to redirect the https pages or is their a better way to stop them from being indexed?

Thanks,
Chris