| 4:22 pm on Dec 22, 2010 (gmt 0)|
My old sitemap has all the old and new links being indexed by Google. I have changed urls of my few pages which were already indexed by Google and put the canonical tag to tell Google to consider only my new pages and ignore old pages though my old pages have been 301 redirected to new pages as well. I have added new canonical pages URLs to sitemap. Should i remove the old indexed URLs now immediately or not now and do after some time? Give me advice please to secure my old indexed pages in Google search results.
| 6:11 pm on Dec 22, 2010 (gmt 0)|
An XML Sitemap should not include URLs that 301 redirect - so yes, remove them. Those old URLs should also not appear in your internal links anywhere. This will help economize on Google's crawl budget and your new URLs should stabilize faster in the rankings.
| 9:55 am on Dec 23, 2010 (gmt 0)|
Thank you very much for your advice, i will do that.
| 4:48 am on Feb 1, 2011 (gmt 0)|
It has been over a month and Google has reindexed very few 301 redirected pages. Site traffic is also getting down dramatically and by site: operator i have only 10,000 pages left out of 30,000.
| 1:05 pm on Feb 1, 2011 (gmt 0)|
sitemaps & 301 redirects are useful but your website needs a healthy backlink profile for you to get properly indexed & ranking. i think you might want to revisit your backlinks and make sure you have enough.
| 4:22 am on Feb 2, 2011 (gmt 0)|
Thanks for the reply, Goodroi. Do i need healthy backlink profile for my already indexed pages to be recrawled? Does Google not automatically recrawl and reindex already indexed pages having 301 redirects? I am losing daily 500 to 800 indexed pages by checking site: operator.
| 5:04 am on Feb 2, 2011 (gmt 0)|
You want to be clear about the difference between getting re-crawled and having those pages stay in the index. Google crawls a lot of pages but it doesn't actually keep them all in the index. And you'll often see the site: operator results go up and down a lot over time.
But the short answer is that the more backlink power is flowing around your pages, the more likely they are to stay in the index.
| 5:19 pm on Feb 2, 2011 (gmt 0)|
The canonical tag will not be seen on old pages since they respond with 301 redirect.
|...put the canonical tag to tell Google to consider only my new pages and ignore old pages though my old pages have been 301 redirected to new pages as well. |
and then subsequently in a later post:
|My old sitemap has all the old and new links being indexed by Google. |
|I am losing daily 500 to 800 indexed pages by checking site: operator |
The emphasis above is mine. When changing URLs, this behaviour with regards to the number of indexed pages is expected. Daily drop in number of indexed pages is most likely the sign that Google is picking up your 301 redirects.
When you replace a number of old URLs with new, then the common behaviour with regards to a number of pages indexed by Google is:
1) During the crawl, Google finds new URLs on the site (and you will often see increased crawling). Usually, new URLs will be added to index (cavetat: for simplicity, I am here ignoring dupes and low value pages angle)
2) Initially, the number of Google indexed pages will increase because for a period of time Google will have both, the new URLs (that it just discovered on the site) and the old URLs that are doing 301 redirect, but have not been re-crawled yet (hence 301 redirect is not yet seen by Google)
3) In some cases the number of URLs in index may come close to double to what it had before. The site will also experience rise in reporting of "duplicate titles" and "duplicate description" in WMT as both, the old and the new page will now be shown with the same title/description
4) As Google started to re-crawl the "old" URLs that are now serving 301 redirect, the number of Google indexed pages will start to drop sinice Google will start to remove these from its index (and also, the number of duplicate title/description will start to fall too).
Hence Google's "site" operator behaviour you are describing seem normal to me. But to get a better picture on what is happening when doing such change, you really should monitor the following:
a) before the change, how many indexed URLs were reported by Google
b) once the change is implemented, it is likely that a number of indexed pages reported by "site:" operator will increase, so I would watch at what time a number of indexed URLs peaked
c) when the number of indexed URLs started to drop, the rate at which it is dropping. Also, is the drop below the number of indexed URLs you had originally (at step a, before the change)
d) if so, is this drop of reported number of indexed pages expected for some reasons? For example, if the new URLs also fix some canonical issues that you had with "old" URLs then I would expect that the number of pages reported by "site" operator drops below to what it was before the change.
e) What is happening with ranking of your important pages that bring in the traffic during this change (I would expect that at some point their URLs are replaced with new URLs and hopefully the ranking remains where it was or bounces back soon).
You must bear in mind that, depending on the size of the site and the crawling frequency, it could take some time for all old URLs to be re-crawled and 301 redirect discovered. Monitoring appearance and disappearance of duplicate titles in WMT is an indicator on when the new URL has been discovred, then subsequently, whether the old page has been recrawled and its 301 redirect discovered.
Cavetat: the above assumes the only thing that changed are number of URLs within the site (i.e. that site structure, interlinking, anchors etc did not change)