Permanent redirect, 301... should the new url replace the old?

[fretfull]

I'm wondering what happens when a 301 redirects an old URL to a new one. The page content is the same, it's just the URL that changed. I did this a while ago. The old URL has been re-crawled a couple of times. I expected the old URL to be removed from the index and the new URL to show up in search results instead. This isn't what's happened so far. Both the new and old URLs are in the search results. The old URL has a higher page ranking. Google has noted that the two URLs have duplicate metadata. If I remember correctly, it has also noted that the old URL is a redirect. Are both URLs going to be in the search results permanently or will things sort out in time?

[dennisjensen]

Hi,

"If I remember correctly, it has also noted that the old URL is a redirect". Use a http response test to varify, if your 301 works as intended.

G will obviously need some time to adjust, but if they have noted your redirect, but still ranks the Url. Makes no sense.

[not2easy]

1. Can you still access the "old" URL?
2. Have you added the "new" URL to your GSC?
3. Which URLs are on your sitemaps?

[wavingdave]

I have this exact issue with several key pages at the moment.

I put 301s in place last September, while trimming similar content, and they work fine with all browsers but it's still the old URLs showing in the Google SERP results (1st page, #8) with the new URL content nowhere to be seen. Frustratingly, it's a daily report and uses the date in the page title (URL stays the same)... so you can instantly see that it's out-of-date in amongst competing pages that are just as clearly current.

FWIW, DuckDuckGo has followed the 301 ok and has our new page at #7 for the same phrase.


If I use INSPECT URL on GSC :-

- With the new URL, it all looks good with GSC reporting under "Coverage" that the URL was found via our sitemap.

- With the old URL, the information under "Coverage" is different (showing N/A next to sitemaps and a different last crawl date).

- Using TEST LIVE URL, with the old URL, GSC clearly follows the redirect as I see the content from the new URL!


So, it looks as if it's the Smartphone Googlebot that's ignoring the 301, even though GSC's TEST LIVE URL follows it correctly.

[wavingdave]

Actually, thinking that through... it's more likely to be an issue with Google's cache (of the old URL) isn't it?

[not2easy]

If your 301 was implemented correctly, the old URL is the new URL. If it was not implemented correctly it can be seen as a temporary change, which would hinder indexing.

Visit the Apache Forum [webmasterworld.com] if your site is hosted on an Apache (UNIX) type server. For Windows servers see the IIS/Windows Servers [webmasterworld.com] Forum to check your implementation. Or use our Search to look for "301 redirect". If you have tools to check your live headers you can see it. If you use developer tools, available for most browsers, you can see whether it is a 301 or 302 response.

If you clicked something in ControlPanel to handle your rewrite you may have a 302 (Temporary) response. That is the Apache default unless specified 301.

[lucy24]

it looks as if it's the Smartphone Googlebot that's ignoring the 301

Nobody--not even google--can ignore a 301. Once a 301 response has been received, there are only two options: follow the redirect, or go away. Google currently follows redirects almost immediately, just like a human browser, unless coincidentally it has already recently crawled the new target URL. It is possible that the two googlebots share resources, so the mobile won't follow a redirect if the new URL has recently been visited by the non-mobile. And, presumably, vice versa.

Check your raw logs to verify that all requests, all the time everywhere, meet a 301 response when they request the old URL. In general, any such 301 received by the Googlebot will be followed within a second or two by a request for the corrected URL.

:: detour to raw logs, leading to intriguing discovery that Google likes to request groups of URLs that it already knows will be redirected, as in “GET /oldurl1.html” and “GET /oldurl2.html” not followed by “GET /newurl.html” (which it already knows is the target of both oldurl1 and oldurl2) ::

[fretfull]

Thanks for the replies everyone.

I'm pretty sure that the 301s are implemented correctly. If I go to the old URL in a browser, it redirects me to the new URL.

As another poster mentioned, this only seems to be a problem with Google. Bing for example has effectively replaced the old URL with the new one. The new URL has replaced the old URL in page rankings. It will still show me the old URL if I search explicitly for it, but otherwise I don't see it.

This has me so bewildered. I thought it would be an easy process. For the other search engines it has been. But Google seems to be sticking with the old URLs and ignoring the redirects.

[wavingdave]

Nobody--not even google--can ignore a 301.

lucy24 - you're quite right - my bad; hence my follow-up reply.

So presumably Google is pulling the old URL (and content) from it's cache to populate the SERP?

I see the old URL, with the old snippet/title in the SERP... and when I click on that the browser redirects correctly straight to the new page.

Just to add to what I posted above, Bing also has our new page at #7, same as DDG, Ask (does anyone ever use that any more?) has the new URL at #8. It's just Google holding onto a page that's been gone 5-months.

It's extra strange as the search term is very much from folk looking for today's info; content from 5 months ago is genuinely irrelevant to a searcher - yet there's our old, irrelevant and inaccessible content on the 1st page of Google.

I know I can't provide specifics here, but if any Googler sees this it'd be good to get their take on it?

[tangor]

It will still show me the old URL if I search explicitly for it, but otherwise I don't see it.

Does the old url still exist, or have you deleted it? Inquiring minds want to know.

All too often folks have two pages in existence and from that many problems can follow. ALWAYS DELETE the page which is the target of a redirect. General housekeeping ... and a sure way to prevent duplicate content issues.

[wavingdave]

Does the old url still exist, or have you deleted it? Inquiring minds want to know.

In our case, the old URL has been 301-ed since September, the page is gone, and the only place the title and snippet exists is on page 1 of the "relevant" Google SERP. As I said above; you can click the old URL in the SERP and the 301 takes you straight to the new page.

Happy to message the specifics to any of the senior members who've taken an interest here - are there any resident Google-ists?

[lucy24]

url != physical page

All URLs exist; you see it in logs when Google is checking for 404s by asking for long-string-of-nonsense.html. The only thing that matters is whether a given URL leads to a 200 response. If a redirect from /olddirectory/ to /newdirectory/ is working, it makes no difference at all in the world to anyone anywhere whether /olddirectory/ still physically exists on your server; nobody has any way of knowing it's there. If a malign robot asks for a /wp/ file and you manually serve it a 404, it makes no difference that the requested file actually exists on your server and is readily available to legitimate users. Et cetera.

Yes, OK, if you have vast amounts of old material cluttering up your server, it may take a picosecond extra for the server to locate the current material. But the search engine doesn't know that.

[fretfull]

lucy24, I've been checking the raw logs. Google does seem to be following the redirect. I get a hit on the old URL by googlebot. It gets a 301. One second later, I get a request for the new URL. So googlebot is processing the redirects but in Google search both old and new URLs are still showing up. Old URL still ranking higher than the new URL. I don't understand why this is happening. From everything I've heard from Google reps and others, the 301 should give the old URLs pagerank to the new URL. The new one should replace the old.

[lucy24]

I don't understand why this is happening.

Nobody does. That's why Google is Google while you and I are you and I. But you're doing what you're supposed to be doing, so all you can do is wait. Day by day, new URLs will replace old URLs.

[tangor]

What you see in the serps is NOT real time. G has to update all their data centers and, for the most part, they error on the side of caution in dropping links even when given a 301. IN YOUR CASE it does not matter which ranks higher as if a user clicks the OLD link they get the new one via your redirect. Chalk it up as having TWO listings for the price of one and don't worry about it.

G will eventually update, next few days, weeks, or even a month or three. As long as your REDIRECT is correct you don't care which url is listed.

[fretfull]

It's does matter. Since Google seems to be splitting up the page rank between the two URLs. After all, they have the same content and metadata. Google penalizes a lot for pages that share nothing but the same metadata. I rather have 1 link ranked high up than 2 links much further down. While my new URL is rising, my old URL is falling. It's nowhere near where it was before I did the redirect. I thought that was the whole point of the redirect. To keep that from happening.

It's been more than a few days or weeks. I don't want to measure the time in terms of months. The other search engines adjusted to the redirect essentially right away. One day I searched and the old URL was there. The next day the new URL was where the old one was.

[tangor]

An observation based on a flawed viewpoint. The NEW url is just that "NEW" and starts in the same place all new urls start: at the bottom. The old link (not yet removed from the serps) is the old rank and losing it fast as the new guy sucks it up. TRAFFIC POSSIBLE from both, however. It is what it is, but is NOT magical or instantaneous or starting from parity. The new guy is a new guy link based on an old link the webmaster just informed the SE is no longer valid.

The other way to look at things is DO NOT CHANGE YOUR URLS unless absolutely necessary! Change the content, change the format, change the coding, but leave the URL alone! G (or any SE) loves content change, but every single one of them will take a second or third look at redirection, particularly if there are a LOT of redirects.

And guess what (empirical observation at work), any time a once ranking page gets redirected to a new url it never gets all the ranking back. One way to think about that is getting bumped out of the line trying to get past the bouncer at the destination you are trying to enter. You have to have some extra very important special stuff to resume your place in the queue. Guess how often that happens!

Other aspect of redirect is it has been ABUSED by webmasters over the years. This has created a minor (repeat MINOR) negative effect to be overcome. (g: What? You didn't do it right the first time around and you're making me change things up NOW?) Bing not so much. :)

Meanwhile, one way to force things is to (sadly, can't do it now for the old url in OP) is to kill all SE caching ... I suspect many will speak out against that. Me, I want every index to be BRAND NEW...

[buckworks]

I put 301s in place last September, while trimming similar content, and they work fine with all browsers

It's not clear to me ... did you actually test the 301s with a tool, or just try them "by hand" and they seemed to work right? A tool might reveal something useful.

Google is definitely a slower learner than it used to be. Do everything you can to send stronger, clearer signals.

Are the 301s single-step redirects or are there chains of redirects in the mix? Single-step redirects are preferable.

Are there variants of the old URL in circulation such as with/without .www, /index.html, etc.? If yes, make sure they too are redirected to the new URL. Again, single-step redirects are preferable.

Don't just depend on redirects, though. Are there any links out there that still point to the old URL? Go on a mission to update whatever you can. Make sure that the links under your control are up-to-date, linking to the new URL, one exact URL only, no variants.

[wavingdave]

It's not clear to me ... did you actually test the 301s with a tool, or just try them "by hand" and they seemed to work right? A tool might reveal something useful.

Tested with several tools, headers inspected, single-step 301 from the "old" https/www URL to the "new" https/www URL, nothing complicated and no content in place on the server that could have been served to the old URL. 301 picked up and handled fine, immediately - back in September, by Bing and DuckDuckGo...

...and over 5 months after the 301 was set-up... Google has finally (today), dropped the content from our old page and decided to show content from our new URL in the SERPs.

I have to say I'm with fretfull on this; it absolutely was an issue for us as the information is time-critical and our users want today's info. As I mentioned above, the date (of the info) is in the page title and H1 (and hence in the snippet on the SERP) so exactly nobody would have been interested in clicking our result on Google's page while it said September 2018 on it.

I'm amazed we've only lost one position on the SERP after all that time with an irrelevant snippet but guess I should be grateful for that!

fretfull - any progress with yours?