Migrating a 6-yr-old website with 8k pages - 180 day update

[elos42]

I put in a request to migrate our website (indexed on Google News).

It had both http and https indexed pages, around 8k of them (duplicate content).

You can see some of the data -- such as the pace of migration/re-indexation here: [productforums.google.com...]

It's been 180 days since I put in the request and there are still around 42 page on the old domain that are not migrated yet to the new domain.

I keep track of this by searching for ' site: old.domain -new.domain'. This query gives fewer and fewer results every day, and has fallen to 42 today. When I do a search using the subject lines of these pages, I get results that are indeed on the old domain.

For all other pages (thousands of them), I get search results from the new domain, and nothing from the old domain on a regular search. But if I add ' site: old.domain ' to the query, then I get the same results from the old domain as well. But if I pull up the google cache for those pages, it shows 'this is Google's cache for the page https:// new.domain/uri. In other words, those pages seem to have migrated fine.

My question is, isn't the 180 days a real deadline, or is it just an approximate estimation of how long they take?

We are really hit hard by this migration as the new domain doesn't rank well. Smallseotools shows domain authority of 1.00 for the new domain and 30.27 for the old domain. Moz rank for the old domain is 3.97.

Funny thing is that the old pages seem to rank fine under the new domain. I think Google's simply transferred the reputation or rank of the old urls to the urls. But the new domain doesn't rank anywhere close to the old one.

I still have the option of taking off the 301s and canceling the migration request and moving back to the old domain. But I want to see how this goes since I've already invested 6 months of our company's earnings in this move (more or less).

Has anybody had any experience with such a migration in recent time? What should I expect now? Should I hang in?

[edited by: goodroi at 11:56 am (utc) on Aug 22, 2017]
[edit reason] Fixed formatting [/edit]

[keyplyr]

Hello elos42 and welcome to WebmasterWorld [webmasterworld.com]

I put in a request to migrate our website

You put in the request to whom?

Yes, 180 days should be enough time for new pages to have been indexed, but you do have a lot of pages (8k.)

But if I add ' site: old.domain ' to the query, then I get the same results from the old domain as well. But if I pull up the google cache for those pages, it shows 'this is Google's cache for the page https:// new.domain/uri. In other words, those pages seem to have migrated fine.

Seemes like everything is working as planned.

Sorry your new pages are not ranking as well as the pages at the old domain. Could be a backlink issue. As long as your 301 redirect is doing its job, everything should work out.

It's possible the old site metrics you are comparing your new site to are from an earlier ranking that Google has now updated.

[elos42]

Thanks a lot for your help.

By putting in a request, I meant I put in a request to google for 'change of address' in search console.

I was wondering if I should redirect the 404 pages of the old domain to the homepage of the new domain, or will this be considered manipulative by Google? It's primarily the 404 pages that have not 'migrated' to the new domain name in Google's index.

The content referenced by most of these 404 pages don't exist. Some of the content, for example, were news stories that were withdrawn on the request of the news source. (where these exist, I've put in an Nginx rewrite rule to avoid the 404 errors.)

This seems to be holding the whole process back (for example, if I go to 'site settings', it shows 'some settings for the domain are not available due to a pending address change request. I used to think the address change request would be processed in 180 days.)

As for ranking, every new news-article on our website used to come up on the first page of a relevant search on Google News while we were on the old domain.

On the new domain, almost none of the articles come up on the first page, and most are to be found on the second or third page. There seems to be a domain authority/trust issue. I'm assuming the domain authority will get transferred to the new domain eventually, but it looks like 180 days is not enough.

[keyplyr]

I was wondering if I should redirect the 404 pages of the old domain to the homepage of the new domain, or will this be considered manipulative by Google? It's primarily the 404 pages that have not 'migrated' to the new domain name in Google's index

No, don't try to manipulate this.

The 404 tells Google the document no longer exists. This is working the way it should.

If you did the 301 redirect properly then you are done. Leave it alone and let the process continue.

[elos42]

Thanks. That was indeed my first instinct. I am assuming these pages too will be dropped. I can see a decline of around 1 or 2 pages under indexation under the old domain every day. In another two weeks, all pages still under the old domain should disappear.

[lucy24]

If it's feasible to do so--with multi-K pages it may not be--see if you can return a 410 instead of a 404 for any pages you've expressly deleted. This must of course happen at the old domain. Most search engines don't care, but Google really does seem to stop crawling sooner if it meets a 410.

The problem with a 404 in this situation is that you've probably got a global redirect
example.old/any-url
>>
example.new/any-url
and then the search engine won't get the 404 response until it arrives at example.new. (The same thing happens when google requests a garbage url testing programmatically for “Soft 404”, but in practice there's nothing you can do about that.)

[elos42]

You are right. I have a global redirect.

So the request for the old domain will get a 301 first, and then the new request (after the 301) to the new domain will be met by a 404. Would this be an issue? I mean, would google think -- it says moved permanently, but I can't find it there!

I relooked at the 404 pages. There are about 20 in all. 12 or so are related to a permalink structure change, and were fixed with Nginx redirects.

3 were related to genuinely dropped pages and the remaining 6-7 were related to a test website. (The test website was 'test.old.domain'. It was part of a testing process done about 1.5 years ago (didn't even realize they got indexed. They were up only for a few minutes.)

[keyplyr]

Google really does seem to stop crawling sooner if it meets a 410

Not in my experience.

14 years now and Google still asks for a page I had up for about 2 hours, then removed. Been using a 410 for close to 6 of those years. Must be a backlink somewhere.

[elos42]

I've put a 301 from test.old.domain to the corresponding production page on the new.domain. That leaves only 3 pages or so of genuinely missing content with soft 404. Let's see what happens.

[SEchecker]

Request an address change.

Use the Change of address tool when your site move entails a domain or subdomain change, such as changing from http://fish.example-petstore.com...

...to http://example.com or http://example-petstore.com.

Note: The tool """does not""" currently support the following kinds of site moves: subdomain name changes, protocol changes (""" from HTTP to HTTPS"""), or path-only changes.

[edited by: Robert_Charlton at 6:17 am (utc) on Aug 24, 2017]
[edit reason] Delinked example urls and fixed formatting [/edit]

[Robert Charlton]

Mod's note: Adding link to information on use of Google's Change of address tool cited by SEchecker above...

Use the change of address tool
Search Console Help
[support.google.com...]

[keyplyr]

I believe elos42 did that.

I put in a request to google for 'change of address' in search console

[elos42]

Yes, i did that on Feb 22, and it's still under process.

[SEchecker]

1. The G forum link in elos42 post doesn't work for me....

2. Its not clear to me what elos42 actual change was: http.old > https.new or https.old to http.new is it just sub-domain change or not?

3. As I understand it, elos42 just changed sub-domain name like xxx.domain > yyy.domain

4. Is the duplicate content solved? If not that could be a ranking drop factor tho...

5. I can tell from my experience after changing to new domain lost about 30% of domain weight, never recovered (but all page indexed tho and re-directions work properly)

6. Did you check if your old.domain is still active in search console stats... like still receiving clicks?

7. Are this 42 not migrated pages from big value to you? Are they good performing or just some random content? Do they have maybe spammy backlinks pointing to them? Migration to a news site for sure triggers G to recalculate domain weight, age and backlinks and does recalculate SERP rankings...

[SEchecker]

I just want you to be sure you understood this sentence:

"The tool """does not""" currently support the following kinds of site moves: """subdomain name changes""", protocol changes (""" from HTTP to HTTPS"""), or path-only changes."

Just to be sure :-)

[elos42]

I'll try the link again

[support.google.com...]

There was duplicate content on the old domain as both http and HTTPS versions were in the index, though the regular version almost never turned up in search results. The HTTPS version was put up about four months before the migration to the new domain. At present, both http and HTTPS versions are redirected to HTTPS version of the new domain.

Despite the duplicate content, the old domain used to rank very well, and was on Google news for about five years prior to the migration.

Subdomain issue is not a big deal and accounts for only 7 remaining pages.

My worry is that the new domain has no page rank or domain authority yet, but the old domain ranks well according to page rank and domain authority testing tools. I'm assuming these tools get their signals from Google and Google's yet to transfer the domain authority.

[Robert Charlton]

elos42, a bunch of thoughts that come to mind...

You don't say exactly why you redirected your site... whether it was just an http to https migration, or whether there was also some site redesign as well... and if so, you don't say how much it was changed and in what sequence.

We don't have, eg, a clear timeline of your move... say the timing of your domain change in relation to when pages were removed. Was this done all at once, or was there a sequence?

I'm guessing also that you didn't use a tool like Majestic to determine what pages had good inbound links that were worth preserving. I should note here keyplyr is absolutely correct... that you absolutely do not want to be redirecting the 404 pages of the old domain to your new home page. This is explicitly something that Google warns against, and it would be seen as manipulative.

While your global redirect is preventing any individual page control in any case, there are reasons why you might decide to make your redirects more granular. If those 404 pages were destinations for important inbounds, you might want to look into redirecting the links to related category pages that would in turn link to similar articles... perhaps even preserve the url and indicate on the page that the article is gone, but suggest that these recommended articles you're now linking to might be helpful. User experience would be an important consideration here. Such precise redirection is difficult to do if you preserve the global redirects.

Also, something I've run into that's difficult for some webmasters to understand immediately... when you 301 redirect page A to page B, be aware that all of page A's content and navigation ceases to exist. Because it disappears, your new site's nav structure now depends on what's on the page Bs of the new site. If there's been any changes in the site, how they affect navigation must be carefully considered.

Was there any navigation on these dropped 404 pages that was important to your site structure? News sites very often depend on links to individual articles which then in turn link to related articles, and those may be links that were keeping some articles on page one.

Another consideration is what happened in Google's valuation system when you had pages that transgressed sufficiently that you were asked to remove them. Were they dupes? If so, how was Google regarding the links to them? Was there any penalty? How would Google regard a new page with switched content using the same links? If Google is using something like Word2vec in its link valuations, that would likely change when you changed the content of the page.

Also, what was the quality of your inbound links? I ask, because when you make site changes like this, Google is likely to take a fairly deep look at the site, and some revaluation is likely.

One more thing that raises some questions for me...

I've put in an Nginx rewrite rule to avoid the 404 errors.

I'm not experienced with Nginx, and don't know how it might mix with Apache that I assume you're also using. Are all these 301s? What about things like order in .htaccess?

Run your 301s through a header checker to see that they're all correct and perform as desired. And have you gotten some old inbound links changed (not too many, which might suggest that you control them)? Have you gotten some good quality inbounds to the new urls. These might help suggest to Google that the site has a following, and that the site changes work for that audience.

[elos42]

You don't say exactly why you redirected your site

The site was started as a hobby, with a cheap domain name that had practically nothing to do with the content. However, over the years, it started attracting a decent amount of traffice, going all the way up to 900,000 PVs per month about a year ago.

So I decided to invest into it, and bought a domain name that made more sense, was relevant, and would ensure that people would take the site more seriously compared to the earlier domain name.

whether it was just an http to https migration, or whether there was also some site redesign as well... and if so, you don't say how much it was changed and in what sequence.

It was a simply one-to-one migration -- only the domain name was changed. The wordpress theme was changed, but the permalink structure, navigation etc was exactly the same. I had read up before doing this, and that's one reason why the content was not touched.

We don't have, eg, a clear timeline of your move... say the timing of your domain change in relation to when pages were removed.

In fact, out of the 8,000 pages, only 3 pages were actually removed, and that too over the last say 3 years or so. The other 404 errors were related to a legacy permalink structure dating back to about 1.5 years. I had put in 301s to take care of the permalink (url structure) changes for a year or so, and assumed that that would have been enough. In fact, except for a couple of dozen pages, the permalinks were indeed updated on Google as well. Perhaps the stragglers were probably the least accessed, least popular pages, and that could be why those old permalinks were not updated in Google.

A good timeline of the whole episode, with graphs showing the progression of index changes etc can be seen here. [support.google.com...]

I'm guessing also that you didn't use a tool like Majestic to determine what pages had good inbound links

The inbound links have been 'carried over', going by the Search Console data for both websites/domains. Please see the above link for exact data.

If those 404 pages were destinations for important inbounds, you might want to look into redirecting the links to related category pages that would in turn link to similar articles... perhaps even preserve the url and indicate on the page that the article is gone,

These were amongst the weakest pages out of the 8,000 and very unlikely to have been linked to. What happens is sometimes, a company may 'withdraw' a statement or, in one case, there was a staff error, which forced us to 'recant' a news item. But as I said, these three stories were not a big deal from a user experience perspective.

Also, something I've run into that's difficult for some webmasters to understand immediately... when you 301 redirect page A to page B, be aware that all of page A's content and navigation ceases to exist. Because it disappears, your new site's nav structure now depends on what's on the page Bs of the new site. If there's been any changes in the site, how they affect navigation must be carefully considered.

It's a one-to-one migration, and the structure and content of the website -- as a bot would see it -- would have remained unchanged.

Another consideration is what happened in Google's valuation system when you had pages that transgressed sufficiently that you were asked to remove them.

Google never asked us to remove anything. So we're safe there, I guess.

Also, what was the quality of your inbound links? I ask, because when you make site changes like this, Google is likely to take a fairly deep look at the site, and some revaluation it likely.

within Google Webmaster Tools, when you check on 'Links to your website', all the links that used to point to the old domain are now shown as pointing to the new domain via an intermediary link (intermediary link is the old domain url which is now under a 301 redirect.)

The biggest provider of inbound links is the old domain, which is probably not surprising given the 301s. Out of the total 170k links, 144k are from the old domain.

Have you gotten some good quality inbounds to the new urls. These might help suggest to Google that the site has a following, and that the site changes work for that audience.

The number of links pointing originally to the new domain would be about 5% of the number pointing to the original domain.

[martinibuster]

Smallseotools shows domain authority of 1.00 for the new domain and 30.27 for the old domain. Moz rank for the old domain is 3.97.

If you're worrying about Moz DA and MozRank transferring over, that's just an issue with Moz updating their statistics and has nothing to do with Google at all.

Don't worry about Moz DA (unless you're selling links).

[lucy24]

Back to those 404/410s ...

There are about 20 in all.

Just 20? In that case it is perfectly feasible to hard-code the individual 410 responses at the old site. Unfortunately you will now need to include the same code on the new site--where those URLs never existed--because by now, search engines probably know about them. Pity.

genuinely missing content with soft 404

Your global 301 > 404 isn't a "soft 404". It would only count as a soft 404 if 404s on the new site all globally redirected to the same URL. Which I hope they don't.

You can say a lot about google but nobody ever said they were stupid. If a 301 target involves a different hostname and/or protocol, they understand what's going on.

About Google giving up faster: Googlebot's overall 410:404 ratio is about 8:7. bing/msnbot's is closer to 5:1 (slightly more 404s, vastly more 410s).

[elos42]

"It would only count as a soft 404 if 404s on the new site all globally redirected to the same URL. Which I hope they don't."

No, they don't. They all point to the 'page you're looking for cannot be found' 404 message on the new domain. All others that used to give 404s have now been redirected to the corresponding production page. In other words, I now have only three urls on the old domain that show 404 errors (after a 301). My worry is whether these three cases will keep the domain migration from taking place -- as it's a case of a 301 followed by a 404. I can, of course, create a new Nginx rule for the old domain that prevents a 301 and introduces a 404 directly, without any redirection. I don't want to do it, as I have a feeling that I should not be handing out any response code other than 301 for the old domain no matter what.

[keyplyr]

"It would only count as a soft 404 if 404s on the new site all globally redirected to the same URL. Which I hope they don't."

No, they don't. They all point to the 'page you're looking for cannot be found' 404 message on the new domain.

Ah em... that's pointing to the same page.

[elos42]

It's not really a page, is it? I mean, the response code is not 201, but 404. This is what curl shows:



curl -IL http://urlinquestion

HTTP/2 301
server: nginx/1.10.3 (Ubuntu)
date: Fri, 25 Aug 2017 03:48:17 GMT
content-type: text/html
content-length: 194
location: https://new.domain/uri
strict-transport-security: max-age=63072000; includeSubdomains
x-frame-options: DENY
x-content-type-options: nosniff

HTTP/2 404
server: nginx/1.10.3 (Ubuntu)
date: Fri, 25 Aug 2017 03:48:17 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
vary: Cookie
expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-cache, must-revalidate, max-age=0

[edited by: Robert_Charlton at 4:49 am (utc) on Aug 25, 2017]
[edit reason] delinked sample urls [/edit]

[keyplyr]

That's your curl generated response header. You probably do not want to serve that to your potential visitors.

There should be a nginx default error page or you may need to create a custom 404 page to use for this purpose. Something with a friendly greeting alerting the user that the document they requested does not exist, but please visit other pages on the website.

Also, To avoid posting active links associated with your web properties, please use "example.com" or wrap the snippet in code tags.

[lucy24]

Ah em... that's pointing to the same page.

I believe he's referring to his 404 page. So long as you're not actually redirecting to the literal URL of the 404 page--which people have been known to do by mistake--all is good.

I have a feeling that I should not be handing out any response code other than 301 for the old domain no matter what.

If a page is gone, it's gone. Nothing wrong with saying so up front. In fact it's almost certainly better than 301 > 404.

[elos42]

I dug a little deeper into the new domain's search console messages, and there are 3,500 404 errors. Basically, these are all related to the old permalink/url structure. This permalink scheme had been changed to the one in use presently, and I had kept an nginx redirect for the old urls in place for a year. It was my understanding that the old url structure was updated with the new in Google's index.

With the change of address, what's happened is that Google is again trying to access these old url patterns and claiming that they have links pointing to them. The links, for some reason, are all internal -- from other pages -- probably from 'related posts' sections of pages.

I am thinking of reinstating the nginx rewrite rules to deal with errors. However, rewrite rules and if conditions tend to slow down the overall performance, but I am not sure I have much of an option here.


Google message:

Error details

Last crawled: 6/10/17
First detected: 6/10/17

Googlebot couldn't crawl this URL because it points to a non-existent page. Generally, 404s don't harm your site's performance in search, but you can use them to help improve the user experience. Learn more

[elos42]

The other option I have is to click on 'errors are fixed' in GSC, and then tweak my robots.txt to prevent Google from trying to index or check those pages again. This has the advantage of being simpler and not requiring a lot of conditions to be added to the server configuration.

[keyplyr]

I do that. It stops the error report, but doesn't stop the discovery. Googlebot never forgets links.

I gave up playing the game. The old archaic links that may possibly carry backlink juice, I redirect to similar current pages.

The old pages with no or little juice that got into the error report often, I robot out.

The rest that show up in the error report I just ignore.

[elos42]

I've done that too. In the last 12 hours, 404 errors have gone from 3,500 to 2,250 or so. Not sure if it will impact the search ranking. Btw, the links that point to these 404 pages are mostly from within the site - one reason I'm willing to just block them all with robots.

On the other hand, writing Nginx rewrite rules will be tough.
The current structure is like this:

author/node-id/title

and the old one (that needs to be redirected) is like this

author_node-id_title

Node id is a number and could be just three digits or up to five digits.

[keyplyr]

Well Nginx is better at handling large numbers of concurrent requests, but IMO Apache sure is easier when it comes this kinda stuff :)