Two URLs access the same page - how to fix

[shaunm]

Hi,

Some of my website's pages are accessible in two ways
Such as
example.com/product-overview.aspx
example.com/product-overview

Will there be any problem because of this? Are they considered 2 entirely different pages even though they are same by SEs?

But only one page is showing up in Google's index which is with '.aspx'


What should I do in this regard? Should I put a redirect in this place?


Thanks a lot!

[FranticFish]

I don't know of any way that you can redirect your .aspx url to your extensionless one. The server knows that the real page is the .aspx version, and it displays that when asked for the extensionless one. So if you then try to redirect back to the extensionless display url you end up in a loop.

I'd...
- change the .aspx version file name (product-overview2.aspx will do),
- get your url rewriting to look that up instead when product-overview is requested, then
- remove product-overview.aspx from Google's index by blocking it in robots.txt then using the removal tool in WMT

Perhaps you could specify a canonical of the extensionless version as an alternate fix but I don't know how reliable that is having not tried it myself.

[lucy24]

I don't know of any way that you can redirect your .aspx url to your extensionless one. The server knows that the real page is the .aspx version, and it displays that when asked for the extensionless one. So if you then try to redirect back to the extensionless display url you end up in a loop.

The server doesn't know anything. Always remember that. Computers Are Stupid. They just happen to be stupid very, very, very fast.

Yes, google thinks "page" and "page.xtn" are two different things, just as it thinks "directory/" and "directory/index.html" or "www.example.com/filename" and "example.com/filename" are different things.

What should I do in this regard? Should I put a redirect in this place?

YES. Stroll over to the Apache forum and look for a recurring piece of boilerplate about the redirect-to-rewrite two-step.

Short version:

Use mod_rewrite. Or, ahem, IIS equivalent. Someone can translate, if necessary.

REDIRECT unwanted ugly URL to desirable pretty URL, with accompanying RewriteCond looking at {THE_REQUEST}. You only want to redirect the requests that came in from outside, not the ones that result from an internal rewrite. That's how you avoid your infinite loop.

REWRITE pretty URL to real home of content-- which might be the identical address you only just finished redirecting from :)

[shaunm]

@FranticFish

I don't know of any way that you can redirect your .aspx url to your extensionless one. The server knows that the real page is the .aspx version, and it displays that when asked for the extensionless one. So if you then try to redirect back to the extensionless display url you end up in a loop.

Thank you for answering. And I am sorry that if my question have some misunderstandings. Btw, I never wanted the '.aspx' version to redirect to extentionless. And you are wrong the server displays both version when requested.

I actually wanted to ask if it is correct to redirect the extensionless to extension one if there is a problem with two versions of the same page. Also, I strongly disagree on your first sentence, yes it can surely be done!

Thanks

[shaunm]

@lucy24

Thank you so much. All I wanted to know is that whether redirecting the extentionless to .aspx will prevent my site from duplicate content or what.

At the same time, I wonder why only the .aspx version is indexed by Google not the one without .aspx. Server displays both version of pages when requested, though.

And I am very much familiar with REWRITE rules :)

Once again, thanks

[Sand]

Google is pretty forgiving when it comes to file extensions these days. They've pretty much got it figured out at this point, but to be safe, it's a good idea to pick one and redirect the other to it. It won't do any harm, and can only help.

As for why only one is indexed right now: it could be any number of things (such as how you're linking to the pages internally), but in the end, Google probably doesn't want to have two pages in its index from the same site that are identical. They know what's going on with the file extension, and have made the choice for you as far as which one they're going to show.

[aakk9999]

At the same time, I wonder why only the .aspx version is indexed by Google not the one without .aspx. Server displays both version of pages when requested, though.

Unless you are using canonical link element, Google has probably decided to pick one version of URL and shows that one.

BTW, it could be someting in your IIS server setup that says "if I get a URL without the extension, append .aspx and forward this to this script".

Or something as above is possibly handled within the custom Rewrite module.

This would result in the exact case you are having: the .aspx page would generate the page content and return it to the client, whilst the client's URL would not change (would remain extensionless).

I seem to remember that IIS 7 for example has Rewrite manager where you can set up rules similar to the above - it is worth checking.

[g1smd]

In general, when two different "path and file"s are available to show the same content, the shorter one should be the canonical URL.

[lucy24]

Google probably doesn't want to have two pages in its index from the same site that are identical

Unless the site name contains an element like "amazon" or "wiki", in which case they will happily show five marginally different versions of the same page, one after the other ;)

[shaunm]

@lucy24

Unless the site name contains an element like "amazon" or "wiki", in which case they will happily show five marginally different versions of the same page, one after the other ;)

I often come across search results like this. What could be the main reason that such websites occupy more than 5 continuous placement/ranking for a single search query?

As far as I know, it only happens with e-commerce websites no matter big or small and also with directories as well.

Thanks,

[shaunm]

@g1smd

In general, when two different "path and file"s are available to show the same content, the shorter one should be the canonical URL.

Thanks. But in my case, it's the opposite :)

@Sand
Thanks a lot!

@aakk9999
Thanks for answering!

it could be someting in your IIS server setup that says "if I get a URL without the extension, append .aspx and forward this to this script"

Can you please tell me what you meant by the above?


Thank you all!

[lucy24]

Oh, are you on IIS? Oops. I'd assumed Apache, and was wondering if mod_negotiation [httpd.apache.org] was being excessively helpful. That is: in a world without search engines or canonicalization it would be nice for the server to read the user's mind and say "when they asked for 'foo' they must have meant 'foo.html' because there's no /foo/ directory". Unlike your average Apache question, MultiViews only comes up about twice a year ;)

[aakk9999]

There is Rewrite module on the IIS server that can be accessed via server admin. Certain behaviour (rules) can be set up there. Rules may use regular expressions. So for example, there could be a rule saying that if Request URL is extensionless, to forward it to the script of the same name with .aspx extension.

This is forwarding and not redirect, this would act similar to server.transfer. I am not saying this is happening, but it is worth checking.

[shaunm]

@aakk9999
Thanks :)

I have access to the Rewrite module.

So for example, there could be a rule saying that if Request URL is extensionless, to forward it to the script of the same name with .aspx extension.

Does that mean the one with extension and the one without extension are both operated by this command? So that I am getting the same page when requested from example.com/text.aspx and example.com/text?

I should seriously check that in that case, thanks for telling me about server forwards. I only knew about redirects.

[shaunm]

@lucy24
Seriously, I don't understand what you were trying to say. Can you please explain?

Also, can you please tell me why pages of the same website are ranking in the first page of search results for a keyword and without any break? Such as ranking in 1st, 2nd, 3r, 4th?

Thanks,

[netmeg]

Also, can you please tell me why pages of the same website are ranking in the first page of search results for a keyword and without any break? Such as ranking in 1st, 2nd, 3r, 4th?

That's a completely different topic than that of this post, and there are already a couple of posts in this forum on that topic. Start here:

[webmasterworld.com...]

[shaunm]

@netmeg
Thanks for that :)
I know it's completely off topic, but wanted to ask since lucy24 mentioned about that.


Cheers!

[g1smd]

In general, when two different "path and file"s are available to show the same content, the shorter one should be the canonical URL.

Thanks. But in my case, it's the opposite

Ah, you're mixing up "paths" and "files". Paths are used "out there" on the web. Files are used "here" inside the server.

You have a "file" called "/page.aspx" and while this can be accessed by the obvious "URL" "example.com/page.aspx" if your site is intended to use extensionless URLs the canonical URL will be just "example.com/page".

[lucy24]

I don't understand what you were trying to say. Can you please explain?

I think it's the same thing aakk said, only we're speaking different languages ;)

[aakk9999]

@lucy
Yes, we were on the same page!

@shaunm

Does that mean the one with extension and the one without extension are both operated by this command? So that I am getting the same page when requested from example.com/text.aspx and example.com/text?

I do not know how your server is set up. But, from your opening post:

Some of my website's pages are accessible in two ways
Such as
example.com/product-overview.aspx
example.com/product-overview

Providing that my understanding is correct, that is, when you enter either of URLs above, the same page content displays and the URL in the address bar shows the URL you have entered, then this would indicate that *somehow* server is set up to internally pass the request to .aspx script.

Now this could happen in several ways:
- Rewrite manager has an entry set up to forward extensionless scripts to the script of the same name with .aspx extension. This means there is a rule set up within Rewrite manager
- there is a "IIS Managed handler mapping" set up to process extensionless requests as .aspx where .aspx variant of the request exists. Handlers map to resources on the Web server. (google "Handler Mappings in IIS 7" for details)
- there is a custom rewrite module in existance that forwards extensionless URLs to .aspx version using the command Server.Transfer (which is used to transfer the execution to a specified script)
- or you may be using ISAPI Rewrite that has this rule somewhere

You also said it happens "for some of your webpages" which to me would indicate there is some custom rule in place if this is not happening for all .aspx pages.

I think you should firstly understand why is this happening before implementing 301 Redirect.

As for the which way redirect should go, whilst extensionless URLs are better (shorter, and technology agnostic), you should also take into the account which format your external links go to, how do you link internally and which URL format Google has in their index.

[shaunm]

@g1smd
Thanks for that. It was not intended to use the extension-less URLs. So I am just going to make the one with .aspx as the canonical :)

@lucy24
I should get used to your language then, thanks ;)

[shaunm]

@aakk9999
Thank you so much for taking your time to explain it to me! It was big, and I very much appreciate your help :)