...put the canonical tag to tell Google to consider only my new pages and ignore old pages though my old pages have been 301 redirected to new pages as well.
The canonical tag will not be seen on old pages since they respond with 301 redirect.
My old sitemap has all the old and new links being indexed by Google.
and then subsequently in a later post:
I am losing daily 500 to 800 indexed pages by checking site: operator
The emphasis above is mine. When changing URLs, this behaviour with regards to the number of indexed pages is expected. Daily drop in number of indexed pages is most likely the sign that Google is picking up your 301 redirects.
When you replace a number of old URLs with new, then the common behaviour with regards to a number of pages indexed by Google is:
1) During the crawl, Google finds new URLs on the site (and you will often see increased crawling). Usually, new URLs will be added to index (cavetat: for simplicity, I am here ignoring dupes and low value pages angle)
2) Initially, the number of Google indexed pages will increase because for a period of time Google will have both, the new URLs (that it just discovered on the site) and the old URLs that are doing 301 redirect, but have not been re-crawled yet (hence 301 redirect is not yet seen by Google)
3) In some cases the number of URLs in index may come close to double to what it had before. The site will also experience rise in reporting of "duplicate titles" and "duplicate description" in WMT as both, the old and the new page will now be shown with the same title/description
4) As Google started to re-crawl the "old" URLs that are now serving 301 redirect, the number of Google indexed pages will start to drop sinice Google will start to remove these from its index (and also, the number of duplicate title/description will start to fall too).
Hence Google's "site" operator behaviour you are describing seem normal to me. But to get a better picture on what is happening when doing such change, you really should monitor the following:
a) before the change, how many indexed URLs were reported by Google
b) once the change is implemented, it is likely that a number of indexed pages reported by "site:" operator will increase, so I would watch at what time a number of indexed URLs peaked
c) when the number of indexed URLs started to drop, the rate at which it is dropping. Also, is the drop below the number of indexed URLs you had originally (at step a, before the change)
d) if so, is this drop of reported number of indexed pages expected for some reasons? For example, if the new URLs also fix some canonical issues that you had with "old" URLs then I would expect that the number of pages reported by "site" operator drops below to what it was before the change.
e) What is happening with ranking of your important pages that bring in the traffic during this change (I would expect that at some point their URLs are replaced with new URLs and hopefully the ranking remains where it was or bounces back soon).
You must bear in mind that, depending on the size of the site and the crawling frequency, it could take some time for all old URLs to be re-crawled and 301 redirect discovered. Monitoring appearance and disappearance of duplicate titles in WMT is an indicator on when the new URL has been discovred, then subsequently, whether the old page has been recrawled and its 301 redirect discovered.
Cavetat: the above assumes the only thing that changed are number of URLs within the site (i.e. that site structure, interlinking, anchors etc did not change)
