Google Spidering, but not Indexing

[lpiracing]

Our site has about 42,000 pages (e-commerce). We recently made a fairly major change in the site which necessitated a change to the URL directory structure. I 301 redirected the URL's and everything seemed to be working fine for about three weeks. I could see Google indexing the new pages in Webmaster Tools and all the indexed pages prior to the change (about 25,000) were redirecting fine. On December 10th, everything changed. Almost like a light switch, many of of the older pages were gone from Google's index.

Since then, the newly indexed pages (which started at about 8,000 pages) keep disappearing from Google's index. The site command shows about 6k right now, but the sitemap in webmaster tools only shows about 1,900 indexed pages. Despite ongoing spidering by Google (which we monitor with Smarterstats and Google Webmaster Tools), the number of indexed pages gets smaller each day instead of increasing! There are no sitemap errors, or any clues I can find in Webmaster tools as to why the site is de-indexing itself.

Some pages are still showing on the first page and the homepage shows up, so I do not think it is a penalty.

The new pages should be under www.example.com/product-type/pc...

The old site was www.example.com/pc/ (all of these pages are redirected)

Our old non E-Commerce site is www.example.com/livper (all of these pages are redirected)

The change needed to be implemented due to the cart system we use.

Anybody have any thoughts?

[edited by: tedster at 5:18 am (utc) on Jan. 5, 2010]
[edit reason] make the domain name anonymous [/edit]

[tedster]

Google has not been behaving in its expected fashion the past few weeks - as other active threads are discussing. There are some guesses that the problems are related to Caffeine's complete roll-out, but no one knows for sure. I do not expect this to be a long-time problem, however.

[devil_dog]

I am having a similar issue...

The similarity with your case is that i too see all pages in site: search but the webmaster tool only shows the number of pages which would actually show up in the organic SERP.

I think my problems also started about December 10th, unsure of exact date cause i noticed it few days after 10th.

my issue is at [webmasterworld.com...]

[Erku]

We are having a similar issue to, we don't know what it is