Little worried about website redesign

[Anon]

I had my online store redesigned and it went up last Friday. Nearly all the URLs changed and I did the 301 redirect for each one. I also submitted a new sitemap within webmaster tools.

I've just noticed a couple of things tonight. Are these something to worry about?

In search results for our best ranked keywords our site is now showing twice - Showing the new link and then directly under the old link (which you click on takes you to the new page). These are taking up position 1 and 2 on average. Cleared history to be sure.

The new submitted sitemap is showing the following in Webmaster Tools:
Web pages
325 Submitted
2 Indexed

Now before I remember our indexed was in the 200's. Could these still be loading?

Thank you in advance.

[Itanium]

You can't do much more than wait. Form what you say, it seems you did everything right.

I switched domains a while back on a rather large site and it was indexed twice too for some time. I even got duplicate content messages in WMT. In the end everything worked out, but it took a few weeks for the smoke to clear.

[not2easy]

Itanium is right, until they crawl the new pages they can't be indexed. By 301 redirecting the old pages you have preserved your old indexed pages' value to bring traffic to the new pages. I would not get rid of the old pages, but you can change the metatag to noindex after the new pages are indexed. Make sure the old pages are not still in your sitemaps. No way to tell without examining your access logs to see which new pages have been crawled.

One thing that might speed up the process is, if you have category pages, or "Department" pages that link to the other pages, you can use "Fetch as Google" in GWT and submit those pages to be indexed.

[netmeg]

When I migrate a site to new URLs, I can generally look forward to 2-4 weeks of bouncing around, duplicates indexed, traffic drops, etc before it all gets straightened out. Just sit tight for a while.

[gameon]

Did you follow Google's official website redesign / move checklist?
Here is the link:
[support.google.com...]

[Kratos]

@not2easy Do you think that once Google has figured out that the old URLs being 301-ed to the new pages are indeed what you're trying to do, does Google provide 100% of the link juice that the old URL had? Or would you still be wasting the 10-15% of link juice that one loses by default with a 301?

I ask because I have had experiences with both sides of the coin. I had a domain that re-gained its rank and even better after it was 301 (old URL to new URL), while I've had other sites that lost about 20% to 30% of its rank (without gaining any extra powerful links and without changing architecture).

Does Google, once it has crawled all 301s to the new site, "think", "Aha! this guy is simply moving his site and the 301s are there to map out the new site again, let's not waste any PR from the URL redirect" or does Google still apply the same PR-flowing dynamic as per a default 301?

[aakk9999]

I agree that Itanium is right, but with regards to this:

I would not get rid of the old pages, but you can change the metatag to noindex after the new pages are indexed.

If the old page is redirecting, the noindex metatag will not be seen. I would not change anything, just leave 301 in place forever.

Here is the timeline of what is happening in cases like this:

1) Google sees new URLs (from the reedesigned site). Google becames very busy crawling all these new URLs. This can later on be seen as a spike in crawling in WMT

2) Google indexes new URLs

3) Existing URLs are crawled at a standard rate, or sometimes even the crawl rate of existing URLs slowed down. So Google now has both, new URLs and the old URLs in its index. This is the stage where you get multiple SERPs entries and duplicate titles/meta descriptions reported in WMT

4) As Google continues to crawl OLD URLs, it sees 301 redirect to a new one. From what I have seen, sometimes it processes redirect straight away (old URL drops from SERPs and duplicate title/meta desc disappears from WMT), but I have also seen that in a number of cases Google does not process 301 redirect immediately - sometimes for a while it still leaves the OLD URL in SERPs despite having it crawled and seeing 301.

5) Sitemap stays with a very few pages reported as being indexed for a little while, even after almost all new URLs have been indexed. If you wait long enough, it will eventually be reported correctly.

In my experience, you need to wait from a few weeks to a few months, depending on the size of the site, how often the site was crawled, how often its pages used to change, etc. Sometimes most of the important pages will revert to new URLs, but there will be some old pages that still keep showing under old URLs for quite some time.

Regarding 10%-15% link juice waste, we have to go on what Google said as any particular number is almost impossible to prove. When the site is redesigned and URL change, then it is often that the other changes done and other factors have much greater impact on what will happen in SERPs afterwards than this 10% - 15%

Better IA, getting rid of other errors, adding some new content, removing some thin pages are usually all part of site redesign which would impact how the site ranks afterwards.

[lucy24]

sometimes for a while it still leaves the OLD URL in SERPs despite having it crawled and seeing 301.

My personal experience, based on moving 6/8 of my site, is that (re)indexing speed is a direct function of page popularity. My most popular page (from the search engine's POV) was reindexed within days, the second-most-popular page shortly afterward... and then things trickled in for weeks or months afterward. (I didn't check closely.)

As always, crawl != index. They will periodically keep requesting the old URLs forever.

[Kratos]

What would then be the turning point at which Google recognizes that a site is being redirected fully?

If you have a site with thousands or 000,000s pages, when does Google recognize that it needs to increase crawling rate to identify the 301s of what it hints (or knows algorithmically) is a site move?

I would guess that if an X number of pages with the top 10% of the maximum page PR of a site / total number of pages of said site are being redirected to a new site and Google has spotted the 301s of the X number of higher PR pages (in relation to the number of all site's pages) in only a couple of days, then Google would understand the 301 (of the higher-PR pages) as a site move and not just a page move; ergo, it will stark crawling faster.

Again, if I had to guess, I'd say they'd at least look at what's going on with the higher-PR pages for strong hints of a site move. Moreover, higher-PR pages are crawled much-more frequently than lower-PR pages, so by taking a hint from the higher-PR pages, Google could acknowledge a site move very fast. Then again, I'm sure that a site with extremely little PR (as is the case of, low-quality sites, scraper sites and thin content links unless they have artificially inflated their PR) would have some serious issues making a smooth move.

I have no doubts Google has a way to hint and thus acknowledge a site move (to then start crawling faster) in its own core crawling algorithm. But lt's remember that Google needs a very good reason to increase its crawling rate outside of what it deems usual for your site.

Any thoughts on this?

[tangor]

The reality is that it takes time.

How long?

That's our (webmasters) question, but there is no answer from G in that regard. I have noticed that Bing is quicker on the uptake, but not by much. Just time.

And keeping those 301's in place until the change is fully effected.

[aakk9999]

And keeping those 301's in place until the change is fully effected.

They really need to be kept in place ideally forever, especially if old pages had external links. Google keeps periodically re-trying pages that have been redirecting to see if they still redirect.

[lucy24]

What would then be the turning point at which Google recognizes that a site is being redirected fully?

I don't think there is one-- unless you're redirecting page-for-page to a new domain, which isn't the situation here.

When g### learns of a new URL, it will crawl it immediately, where "immediately" = within an hour or two. This, again, is based on observations after a 6/8 site move, where their first piece of information was a brand-new front page URL in wmt, linking in turn to brand-new directory URLs and so on. Logs showed a single, comprehensive crawl of the entire site. And that's, ahem, an obscure backwater site. I imagine that if Amazon changed their URL structure, search engines would re-crawl within minutes.

Isn't there a clickbox somewhere in wmt where you say "crawl this URL and all linked pages"?

There is nothing you can do to stop them from requesting old URLs periodically, no matter how often they've seen the identical 301. But if you ever delete a page, it helps if its old URL also returns a 410. It makes one less URL to re-crawl.

[Kratos]

I don't think there is one-- unless you're redirecting page-for-page to a new domain, which isn't the situation here.

Well, my thinking was more on the crawling budget that Google allocates to a site. One of the only good reasons for Google to make the extra effort to crawl one's site faster and deeper would be when 301-ing to a new site (old URL to new URL with same content, nothing changed).

I'm sure Google would be thrilled to update ASAP any major changes in a site's architecture (e.g. full site move) of a high PR site, so I was thinking Google would be able to tell soon enough that indeed a site was doing a full site move ASAP so as to get the new site on its index again ASAP and avoid having old URLs indexed (at the end of the day, it's in Google's benefit to do this so as to have these important/authority sites in their index when they move to a new location). As if there was a trigger when Google would start crawling and indexing a site faster as Google by then knows that the site has moved (or perhaps when changing the URL structure as it could happen when changing a CMS).

My main concern is huge sites like forums (e.g. Webmaster World, which has a homepage PR of 6) or established news sites (which are known for being huge since they update daily with dozens of news).

Again, I'm thinking on a site's crawling budget. A site with a low PR will suffer compared to a high PR site (if everything else equal). By the way, when I say 'high PR site', I'm talking of a site with plenty of high PR pages, since pages have PR, not sites (just saying that so that everyone can understand how I'm using that term itself).

As usual, these are my observations from my experience. The few times I've done a site move of a mildly high PR site (PR5 with plenty of PR 4s and 3s), I was amazed at how fast Google picked up the new domain and dropped the old domain. On the other hand, when moving site with low PR, it felt like Google was at snail speed when updating the changes

[Anon]

Thanks for all the replies. Just an update .. as of today it looks like everything has updated already *phew*.

I wasn't sure whether or not to ask in this thread, but however on search engine results for some reason the URL chain is not showing.

Like for example you search for Dr Smith Whitening Toothpaste

A competitor is:
Dr Smith Whitening Toothpaste
www.example.com > ... > oral care > teeth whitening
Description

While our is just:
Dr Smith Whitening Toothpaste
www.example.com
Description

These worked with our old site but not the new one. Is this something that could still be taking a while to register or is likely we have done something incorrect?

[lucy24]

Generic answer for all google-related questions:

Wait. They'll sort it out.

This applies to all possible questions except the ones beginning "I received a notice of manual action..."