Google Duplicate Content Fix (IIS/ASP Classic)

How I fixed my www/non-www and default content duplicate content woes

[Panic_Man]

Since 'duplicate content' became an issue with Google indexing, there has been some discussion on the effects of this aspect of the algorithm on SERPS. For us, this indentified potential issues with our existing sites - We had duplicate pages indexed in Google with both 'www' and 'non-www' URLs e.g.

http://mydomain.ext/mypage.asp
and
http://www.mydomain.ext/mypage.asp

AS the majority of our indexed pages icluded the 'www', I developed some code that formed part of a 'config' include placed at the start of each of our pages (ASP Classic) that simply detected the lack of a 'www' in the URL of a page and used a 301 redirect to a modified URL with the 'www' added as follows:

if instr(lcase( request.servervariables("SERVER_NAME") ),"www") = 0 then
Response.Status = "301 Moved Permanently"
strNewLocation = "http://www.mydomain.ext" & StripDefaultDoc( request.servervariables("PATH_INFO") )
if len(request.servervariables("QUERY_STRING")) then strNewLocation = strNewLocation & "?" & request.servervariables("QUERY_STRING")
Response.AddHeader "Location", strNewLocation
end if

(Previously posted here [webmasterworld.com])

Recently however, we identified another potential duplicate content issue in the following:

http://mydomain.ext/myfolder/
and
http://www.mydomain.ext/myfolder/default.asp

We needed to redirect [mydomain.ext...] to [mydomain.ext...] anywhere it might occured on the site.

Unfortunately, this problem was a little more tricky. At first, it seemed that ASP servervariables were unable to determine the difference between the two requested URLs. Calling request.servervariables("URL") (or "PATH-INFO") would return "/myfolder/default.asp" in both cases as, when requesting a folder root, IIS would automatically add the default document name onto the "URL" result. With both results being the same, we had thought we would be unable to 301 redirect these requests.

The answer lay in a 'quirk/feature' of the default documents list in IIS. We dicovered (purposefully or accidentally) that IIS has set all the default documents with capital letters e.g.:

Default.asp
Index.htm
etc..

Looking back at the request.servervariables("URL") results the difference became clear:

Requesting http://mydomain.ext/myfolder/ resulted in "/myfolder/default.asp"

Requesting http://mydomain.ext/myfolder/ resulted in "/myfolder/Default.asp"

So long as we made a case-sensitive comparison of the "URL", we could determine when default.asp was included in the URL and 301 to the root of the folder instead with the following update to our existing code:

if instr(lcase( request.servervariables("SERVER_NAME") ),"www") = 0 or instr(request.ServerVariables("URL"),"default.asp") then
Response.Status = "301 Moved Permanently"
strNewLocation = "http://www.mydomain.ext" & StripDefaultDoc( request.servervariables("PATH_INFO") )
if len(request.servervariables("QUERY_STRING")) then strNewLocation = strNewLocation & "?" & request.servervariables("QUERY_STRING")
Response.AddHeader "Location", strNewLocation
end if

This may be useful to you if, like us, you're not sure if this is an actual issue or not (and can't really afford to take the risk!)

[edited by: Panic_Man at 8:13 pm (utc) on Jan. 27, 2007]

[brickwall]

Thanks for this.

In addition, I noticed that in cases where a slash is omitted from a URI pointing to a directory resource such as

http://www.example.com/folder

IIS automatically issue a 301 and redirects to the slashed version of the URI

http://www.example.com/folder/

Is this setup by default or did my host set it up to behave this way?

I would also like to know if
http://www.example.com
is the same as (in the eyes of the se bots)
http://www.example.com/

I looked at the headers and in both cases, IIS responds with a 200

Does http://www.example.com need to be 301'd to http://www.example.com/

[Panic_Man]

I've yet to see two such URL's indexed together, so I can only assume Google's systems have worked that one out.