MSN doesn't update addresses using proper redirects

When using permanent redirects, MSN updates its content but not the URL.

[mvl22]

I've noticed that MSN doesn't detect new URLs when using server-side redirects.

On various sites I've recently rewritten for my employer, I've reorganised the directory structure into something far more sensible. Naturally, like any good webmaster, I've put in Redirect Permanents into the Apache configuration, so that all the links, bookmarks, etc., to the site continue to work.

Google automatically detects the new address and then removes the old one from its index.

However, I've noticed that MSN, when spidering, is picking up the new page content (I can tell, because the browser-bar title is different) but *doesn't* pick up the new address, despite the fact there is a proper server-side redirect.

Can anyone offer any further information about this considerably annoying 'feature' of MSN (and presumably others). This means potentially that it's impossible to get old URLs out of its engine.

[seth_wilde]

Not sure about this one.. so I'll go ahead and bump it to see if we can get you a proper answer..

[jatar_k]

Are they freebie/paid ink/other pages? I would like to know what type of listing they are because that will help better understand the situation.

I am not sure what the best approach for this would be. You could leave the 301's if they are getting the new content. Not ideal but as long as the spider and the users find the pages it is not as bad.

I can think of a few other things but they are more on the untested side.

[mvl22]

They are generally very established not-for-profit sites.

(I would put a URL here, but I have a feeling that's against the board's rules: if not, let me know and I'll post a search result URL so you can have a look at this phenomenon yourself).

Martin

[mvl22]

[meant to add]

.. and therefore 'freebies'.

I plan to leave the redirects list in place, but for one site I can't guarantee this as it's hosted on someone else's server, and they could withdraw it.

[jatar_k]

I worry about messing with freebies too much because I don't know what happens if the listings go 404. Do they give up or do they respider the site?

That is really the question here. If they are good about respidering the site then just let them 404 and they will come back and get them. If they aren't then I am not sure what approach you should take since the 301's don't seem to be adhered to.

[mvl22]

The point is that their engine does not generate 404s: they follow to the new addresses, but list the content found under the old address.

Should I post the URL to illustrate the problem (it's a UK University site, so I'm not trying to advertise or anything)?

[jatar_k]

OK, I get it. I wanted to make sure it was sending the 301 header properly so I checked it here.

Server Header Checker [searchengineworld.com] and it is fine.

So, what I was referring to earlier was what would MSN do if you just had a custom 404 that did the old redirect to another page for the users? Since the 301 doesn't seem to work the only other option I can think of is to 404. The only thing I don't know is if MSN will then come and get all of the new url's by spidering the site.

[jatar_k]

anyone else have any ideas, this is dealing with uk.search.msn.com. I have no other ideas and msn doesn't seem to care about the 301.

[mvl22]

Thanks for taking a look.

Using 404's wouldn't be safe, because other search engines which act correctly would be turned away.

(The same problem also happens on search.msn.com ; it's not just the UK version).

It simply looks to me as if MSN (or whereever the results actually come from) simply has no understanding of 301s, at least for non-paid for sites. This is bad news for the web, especially when people go to the trouble of updating their site with redirects properly.