Years after domain change, old domain still in SERPs

[sangi]

Hi all,

it's been two and (almost) a half years now after I switched from old and quite authoritative domain blue-widgets.com to a new one without the hyphen bluewidgets.com. I did it mainly because I wanted to increase direct visits and, at least from this point, the change was a success.

The website itself is quite large, there's hundreds of thousands of indexed pages in Google.

The problem is, that the old domain name is still live and well in G's SERPs and I can't figure it out why is that and is it a problem (I think it is).

Now, here's what I did when I switched the domain names more than two years ago:

• set up a 301 redirect from blue-widgets.com to bluewidgets.com (in apache's config: Redirect permanent /, checked the response headers many times with curl, it's all good);
  
• in GSC (WMT) submitted a Change of Address;
  
• changed references from blue-widgets.com to bluewidgets.com everywhere I could (mostly in my site).

Interesting thing: more than a year ago I've added the SSL to my new one bluewidgets.com. Everything is good, no traffic changes from Google, everybody's happy. One day I get the message from GSC: "SSL/TLS certificate does not include domain name https:/ /blue-widgets.com/". "That's weird" - I thought, but without any research I just added the SSL to the old domain blue-widgets.com too.

As you see, it's a long story, but here's the final: even today if I do the "site:blue-widgets.com" Google search, I still get hundreds of thousands of indexed pages! What's even more weird, hovering on any of the links in SERPs I see the https version is indexed (say, https:/ /blue-widgets.com/about-widgets etc.).

How is that possible? Is Google broken? Or did I messed up badly at some point?

Thank you all

[aakk9999]

In a hindsight, it could have been better if you redirected bluewidgets.com to blue-widgets.com instead, and caught type-ins in this way (but this is just a hindsight - do not revert now!)

From the site move point of view, it appears that you have done it all correctly. What I am wondering is whether there are some very strong external links going to blue-widgets.com which may influence Google's behaviour? Also the site is quite large so it may take a long time for Google to process it.

With regards to bluewidgets.com (no hyphen), does this site appears in SERP? And where it does, what happens with the same page from blue-widgets.com, is it also there and if it is, which one ranks higher?

[sangi]

In a hindsight, it could have been better if you redirected bluewidgets.com to blue-widgets.com instead, and caught type-ins in this way (but this is just a hindsight - do not revert now!)

Yes, I agree with you on this, aakk9999. In fact I even like hyphened version more, because it's easier to read. But there's no way back.

From the site move point of view, it appears that you have done it all correctly. What I am wondering is whether there are some very strong external links going to blue-widgets.com which may influence Google's behaviour? Also the site is quite large so it may take a long time for Google to process it.

There's no such strong external links that I'm aware of. I guess it's important to note that prior domain change, the website with its old domain was on its sixth year, the new domain just passed its two and a half years. Even when you start typing website address in, say, Google search box or Google Chrome address bar, the old domain suggestion pops in.

With regards to bluewidgets.com (no hyphen), does this site appears in SERP? And where it does, what happens with the same page from blue-widgets.com, is it also there and if it is, which one ranks higher?

Hyphen-less (bluewidgets.com) site appears in SERP and has approximately 2x more indexed pages than the old hyphened one (blue-widgets.com), if you can trust public indexed pages counter ("site:blue-widgets.com" in Google search) of course as the numbers are quite different in GSC. BTW, in GSC indexed pages counter for blue-widgets.com is around 30. I have never seen blue-widgets.com ranking on any of keywords after domain swap.

So yeah, quite a situation. Now I'm thinking about canceling domain change in GSC for blue-widgets.com, turning off all redirects and putting 410 Gone for all indexed pages with blue-widgets.com (and I'll leave some message with link to bluewidgets.com for my two-years-old-change-unaware users). That should do the trick to deindex blue-widgets.com and stop this confusion, right?

[aakk9999]

I personally would not do this as you risk a drop in ranking since in this way you would lose redirect juice from hyphenated onto not-hyphenated.

If visitors are arriving to your site, does it really matter? I know it matters for branding perspective, but even if they click on hyphenated, they will land on your non-hyphenated domain

Also, it would be good to know how many visitors are in fact arriving from SERPs via redirect - you can find this out by inspecting your server logs.

[sangi]

Well, I'm thinking situation like this could possibly create some sort of duplicated content issues in Googles POV. I mean, now there's hundreds of thousands of indexed pages on both domains. If I try to look at this as an outsider - it looks suspicious, even though there's no duplicated content and everything is 301 redirected, url's are cannonicalized.

If I'm right, then I'm probably getting less visitors from Google than I could.

On the other hand, it's been more than two years after switching domains, so I believe all the good juice are transferred already. The risk of going down in SERPs are minimal, IMO.

Just checked the logs - up to 50 visitors/day from SERPs via redirect. Now all I'm hearing in my head is "do it do it do it!".

[engine]

Just checking:
1. Do the files for the urls for blue-widgets.com still exist on your server?
2. Does googlebot keep visiting those urls at blue-widgets.com?

[sangi]

1. Not sure what do you mean by 'files'. It's same VPS (both domains pointing to the same VPS/IP), so files (images, css, js etc) are definitely here. If you try to open blue-widgets.com/css/template.css directly you will be redirected to bluewidgets.com/css/template.css though. Same applies to all files on this server.

2. Yes, googlebot (and others) keep visiting those urls all the time.

[not2easy]

Does the old domain still serve a sitemap? Do the old URLs have a "noindex" robots meta? I ask because it can take a long time for Google to digest changes on very large sites as mentioned, but mixed signals might make it take even longer.

[aakk9999]

Regarding possible duplicate content, yes, there is a period of that and it happens when Google has discovered URL from bluewidgets.com and has not yet recrawled the same URL on blue-widget.com to see the redirect. Until Google requests blue-widget.com, it will not see that it in fact redirects so it will have two pages with the same content.

But if you revert, you will have the same problem, just the other way around, and you say that there are more pages indexed on domain without the hyphen than on the hyphenated domain.

The question about the sitemap from not2easy is a good one!

@sangi, it would be good to serve sitemap on the OLD hyphenated domain, with all URLs in there being WITHOUT the hyphen, and make sure the sitemap does not redirect. This will re-inforce the message to Google that the site has moved.

<added>
Also, there is another important thing:

Interesting thing: more than a year ago I've added the SSL to my new one bluewidgets.com. Everything is good, no traffic changes from Google, everybody's happy.

So you say your traffic did not suffer after the domain change. Do not compromise this just because Google is still a bit confused with indexing aspects!

[sangi]

Does the old domain still serve a sitemap? Do the old URLs have a "noindex" robots meta? I ask because it can take a long time for Google to digest changes on very large sites as mentioned, but mixed signals might make it take even longer.

No, there's no sitemap on the old domain from day one (now I see that was a mistake). The Old urls does not have a "noindex" robots meta as all urls are redirected (301) to the new domain.

Regarding possible duplicate content, yes, there is a period of that and it happens when Google has discovered URL from bluewidgets.com and has not yet recrawled the same URL on blue-widget.com to see the redirect. Until Google requests blue-widget.com, it will not see that it in fact redirects so it will have two pages with the same content.

But if you revert, you will have the same problem, just the other way around, and you say that there are more pages indexed on domain without the hyphen than on the hyphenated domain.

I would agree with you, aakk9999 if not the fact of newly reindexed urls on blue-widget.com with https:// (see OP). I was stunned after I discovered this!

The question about the sitemap from not2easy is a good one!

@sangi, it would be good to serve sitemap on the OLD hyphenated domain, with all URLs in there being WITHOUT the hyphen, and make sure the sitemap does not redirect. This will re-inforce the message to Google that the site has moved.

Now that's an interesting thought. Sounds a bit weird, but definitely makes sense. One thing though - it's been more than two years and in that period there were (and are) many new urls which never existed on the old domain. I'm not quite sure at this point, but it looks like almost impossible to distinguish which urls were and which weren't on the old domain. Having this in mind, if I do this, I would be blindly bombing Google with thousands of urls and wasting my crawl budget, wouldn't I?

P.S. I'm a long time lurker here at webmasterworld, you can't imagine how much I'm enjoying this discussion. Thanks guys!

[not2easy]

I wouldn't put a sitemap on one domain with URLs for another domain. Better to have none on the old domain imo. If you use GSC, you can go to "Sitemaps" and resubmit a fresh one. I have found that to be helpful in getting their attention for changes. You can also try using the "Fetch as Google" for a few of the new URLs and see if there are any hindrances for Google. It will show you if there are blocked resources for example - and ask if you would like to submit it to index.

[sangi]

I appreciate your input, not2easy. A fresh sitemap (on the new domain - that is bluewidgets.com) was submitted from day one too. "Fetch as Google" also looks good.

I think I'll give it a try with the 410 GONE on the old domain.

[aakk9999]

I would agree with you, aakk9999 if not the fact of newly reindexed urls on blue-widget.com with https:// (see OP). I was stunned after I discovered this!

I missed that - you are right, this would indicate the issue could be domain-wide.

Regarding sitemap, submitting sitemap with new URLs on the old site worked for me several times, but the sites were not as big as yours.

On article on Site Move, Google suggests that both sitemaps (with old and new URLs) are submitted on the NEW site:

On the destination site, submit the two sitemaps you prepared previously containing the old and new URLs. This helps our crawlers discover the redirects from the old URLs to the new URLs, and facilitates the site move. [support.google.com...] .

There is a past thread that describes a cache and redirect situation similar to yours, perhaps worth reading.
Google Still Indexing 301 Content In SERPs? [webmasterworld.com]

Regarding trying with 410 Gone, I would do this as a last resort and in that case I would select a handful of pages where you are sure that the old page has been attempted to be crawled by Google and do them first and monitor whether it has any effect (desired result).

[sangi]

Thanks again, aakk9999, these are an interesting reads.

Just noticed one more funny thing in this G-riddle:

newly re-indexed old domain with https (https://blue-widget.com, which supposed to be gone from SERPs) has original (https://bluewidget.com) page titles(!). I include the domain name at the end of almost every page title (for ex.: "Page about blue widget - bluewidget.com"), so I'm 100% confident about this statement.

The more I dig, the more I realize the G is not omnipotent as I usually tend to think.

[JS_Harris]

Google's memory isn't just long, it's permanent. GWT still reports pages that haven't existed since 2006 as being suddenly found yesterday and 404 today. It happened so often in fact that I began suspecting someone was targeting my site maliciously but nope, just a long memory.

[Robert Charlton]

Several thoughts that haven't been discussed come to mind, and there's a lot of conjecture built into this. I'm not completely sure of your timeline, and can only guess about how Google's databases interrelate.

First... are you seeing these at all in competitive searches? ...or are they confined to site searches (which you'd mentioned) and perhaps also to searches for the old domain name?

If the latter... ie, if these are not for competitive searches... here's an old thread that discusses some possibilities relating to a change in what Google showed for domain-name queries....

Domain name replaced in SERPS with alias domain name
June, 2011
https://www.webmasterworld.com/google/4327200.htm [webmasterworld.com]

There was a follow up discussion here as well...

Google showing 301ed URLs in SERPs
July, 2012
http://www.webmasterworld.com/google/4477793.htm [webmasterworld.com]

I don't think that the above is the issue by itself, but combined with several other factors might provide a clue. Like your story, it gets complicated. You'd also mentioned....

My emphasis added...

Interesting thing: more than a year ago I've added the SSL to my new one bluewidgets.com. Everything is good, no traffic changes from Google, everybody's happy. One day I get the message from GSC: "SSL/TLS certificate does not include domain name https://blue-widgets.com/". "That's weird" - I thought, but without any research I just added the SSL to the old domain blue-widgets.com too.

As you see, it's a long story, but here's the final: even today if I do the "site:blue-widgets.com" Google search, I still get hundreds of thousands of indexed pages! What's even more weird, hovering on any of the links in SERPs I see the https version is indexed (say, https://blue-widgets.com/about-widgets etc.).

and also...

Just noticed one more funny thing in this G-riddle:

newly re-indexed old domain with https (https://blue-widget.com, which supposed to be gone from SERPs) has original (https://bluewidget.com) page titles(!). I include the domain name at the end of almost every page title (for ex.: "Page about blue widget - bluewidget.com"), so I'm 100% confident about this statement.

What I'm thinking... and I really haven't nailed this down, but I'm tossing it out as a possible clue... the inclusion of old domain in your SSL/TLS certificate might conceivably have triggered new Google indexing preferences to display https pages if it believes they exist. The indexing preferences are described in the thread below...

Google Will Seek Out HTTPS Pages By Default
Dec, 2015
https://www.webmasterworld.com/google/4782828.htm [webmasterworld.com]

From Google's announcement, quoted in the thread...

....we'd like to announce that we're adjusting our indexing system to look for more HTTPS pages. Specifically, we'll start crawling HTTPS equivalents of HTTP pages, even when the former are not linked to from any page. When two URLs from the same domain appear to have the same content but are served over different protocol schemes, we'll typically choose to index the HTTPS URL if:

It doesn't contain insecure dependencies.

It isn't blocked from crawling by robots.txt.

It doesn't redirect users to or through an insecure HTTP page.

It doesn't have a rel="canonical" link to the HTTP page.

It doesn't contain a noindex robots meta tag.

The sitemaps lists the HTTPS URL, or doesn't list the HTTP version of the URL

The server has a valid TLS certificate.

The above, as I understand it, is for discovery only... it's not about ranking, but it could be affecting what you see in site: search.

I'm thinking that the inclusion of the old domain on your TLS certificate might have created a difficult problem for a multi-index NoSQL database like Google. This, I should add, is speculation, but I feel it's worth considering.

My guess, and this is a guess, is that the quickest fix might be to include a "rel="canonical" link to the HTTP page", which is included in one of the conditions described in the list above. Other conditions might also apply.

[sangi]

JS_Harris, yeah, your experience confirms 'Google never forgets' theory.

Google's memory isn't just long, it's permanent. GWT still reports pages that haven't existed since 2006 as being suddenly found yesterday and 404 today. It happened so often in fact that I began suspecting someone was targeting my site maliciously but nope, just a long memory.

First of all, thanks a ton for your reply, Robert. As always, I'm impressed with your post quality.

Now on your questions...

First... are you seeing these at all in competitive searches? ...or are they confined to site searches (which you'd mentioned) and perhaps also to searches for the old domain name?
If the latter... ie, if these are not for competitive searches... here's an old thread that discusses some possibilities relating to a change in what Google showed for domain-name queries....

Domain name replaced in SERPS with alias domain name
June, 2011
[webmasterworld.com...] [webmasterworld.com]

There was a follow up discussion here as well...

Google showing 301ed URLs in SERPs
July, 2012
[webmasterworld.com...] [webmasterworld.com]

I don't think that the above is the issue by itself, but combined with several other factors might provide a clue. Like your story, it gets complicated.

As I mentioned earlier, the old domain blue-widgets.com does not rank on any queries I know/follow. I'm able to get it in SEPRs only when I do site:domain search or when I'm searching "blue-widgets.com".
The older threads you referred to discusses similar issues indeed. It almost looks like it's an old issue Google's unaware of.

What I'm thinking... and I really haven't nailed this down, but I'm tossing it out as a possible clue... the inclusion of old domain in your SSL/TLS certificate might conceivably have triggered new Google indexing preferences to display https pages if it believes they exist.

I'm 100% sure it is not the case as I'm using two separate SSL certs issued by different entities.

My guess, and this is a guess, is that the quickest fix might be to include a "rel="canonical" link to the HTTP page", which is included in one of the conditions described in the list above. Other conditions might also apply.

It's not possible, I'm afraid, as domain blue-widgets.com is (well, was before couple of days) redirected to bluewidgets.com at VirtualHost level - there's no content, no files reachable via blue-widgets.com directly without being redirected (301). And it was the case from the day one.

Now on what I have done to finally resolve this. It may look opportunistic and risky, but I did it anyway. My logic is it's more than two years after domain change and all the 'good juice' from the old domain should've been passed to the new domain. So I created a 410 Gone redirect to all possible blue-widgets.com queries except the main (index) page - I left the old 301 redirect to the bluewidgets.com intact.

If you now type in blue-widgets.com in your browser, you will be redirected to bluewidgets.com (HTTP header 301), if you'll try to open any other page (say, blue-widgets.com/items), you will get custom 410 page (HTTP header 410 and a "noarchive, nofollow, noindex" meta tag in page header) with the clickable link to the bluewidgets.com.

For last five days Googlebot (and others) have been bumping into the 410 Gone wall. I'm witnessing less and less bot activity every day, Google search counter for "site:blue-widgets.com" have gone down a bit and falling every day. Interesting thing - I got an "https://www.blue-widgets.com/: Increase in not found errors" email from GSC!

Other than that the bluewidgets.com is doing okay, I'm not seeing any changes in terms of referrals from Google yet.

[sangi]

Update:

got two emails this morning, one for https ://www.blue-widgets.com and one for http ://blue-widgets.com:

Increase in “404” pages on http(s)://(www.)blue-widgets.com/

To: Webmaster of http(s)://(www.)blue-widgets.com/,

Googlebot identified a significant increase in the number of URLs on http(s)://(www.)blue-widgets.com/ that return a 404 (not found) error. This can be a sign of an outage or misconfiguration, which would be a bad user experience. This will result in Google dropping those URLs from the search results. If these URLs don't exist at all, no action is necessary.

Google's getting 410 Gone error 100%.

Also, yesterday I went to the GSC and changed the crawl rate to max. To my surprise it did work - googlebot now checking at the rate of 1 page per two seconds almost continuously the whole day.

[Dymero]

sangi, I also have a site I transferred to a new domain a couple years ago. I see you've confirmed that you're only seeing it on domain-specific searches, so out of interest I searched for the old domain and it is still there, but the old domain doesn't show up for competitive searches or even on the old domain's branding (the business name changed, too).

I think this is probably just some oddity in Google and I wouldn't worry about it too much. If everything else is setup correctly, I don't see any way it can hurt you.

[sangi]

Thanks for your input, Dymero.

while you might be right, please check out GSC "Search Traffic > Links to Your site" section. Your old redirected domain is at the top of the list, isn't it? Probably with thousands of links?

Here's how it looks on my GSC account for the website on topic: [i.imgur.com ]

Is it an issue? I don't know. Is it normal and safe to ignore? I don't know too, but I don't think so. To me it doesn't make any sense. Because of this I will keep doing what I already did (see Msg#:4794230).

I'll report back on changes as soon as I'll spot them.

[Dymero]

Actually, no, I don't have the old domain anywhere in that list. I would have asked if you're 100% sure you have the redirect setup correctly, but I'm also reading in a couple places that a redirect can cause this issue. So I'm not really sure.

[Spiekerooger]

Hi sangi,

I'd recommend not to trust the site: operator in google search. Google started to value the user input (in your case site:blue-widget.com as search term) higher than the webmaster input of blue-widget.com (that is the 301 redirect). It makes you wonder if everything is alright but the truth is: everything is fine with your site if - as Robert Charlton asked - bluewidget.com is served instead of blue-wigdet.com on highly relevant search terms.

I have a ccTLD that has never been used in other ways than being a 301 redirect to .com/(country-directory)/ where when I do a site:example.ccTLD search on google I see the same behaviour you have experienced with your site.

Google uses the searchers intent in regard of the domain combined with the indexed content of the 301 redirect goal to not confuse the searcher by displaying a different domain than the one in the site: search used.

[sangi]

Dymero,

yes, I'm 100% sure I've done the redirect correctly (see my first post).

Spiekerooger, thanks for chipping in. Yes, I kind of noticed you can't trust the "site:domain" search for page count, on the other hand it is definitely good for monitoring general trend (up/down in terms of index size).

If everything is okay, I'm really interested how would you explain the "Links to Your site" page at GSC (see imgur link, Msg#:4795223)?

I'm still continuing blocking (410) all the URLs on blue-widgets.com except the index one, which redirects (301) to bluewidgets.com. I'm witnessing drop of indexed pages on "site:blue-widgets.com", number of links at the GSC "Links to Your site" page is dropping also. The main site, that's bluewidgets.com, is not affected in any way (yet).