Google's Site Indexing Seems Stuck

[chari7ma]

One of our client's websites is enduring a Google indexing problem in that the number of indexed pages using "site:<website>.com" varies greatly to that shown in Google webmaster tools. The site is about 2 years old and was steadily growing in Google index size until they implemented a 301 redirect for 80% of their urls. This was done so as to include an ID in the url and speed up data retrieval from the database. At the time the 301 was implemented, about 6 months ago, Google had indexed about 300,000 pages. Today, in Google webmaster tools, there are 1,800,000 pages shown as having been indexed from the sitemap, but only 330,000 - 355,000 ever show on Google.

1. Is there a penalty by Google for performing so many 301 redirects? What is the maximum time period it can last?
2. What can contribute to a site not moving up in index size? Google reports it is indexing over 70,000 pages a day on average.
3. What are some of the penalties that a growing site could catch from Google that might cause this?
4. Are there any techniques to getting a site indexed faster? I see websites that have been only online 6 months, have low pagerank and are over 1,000,000 pages.

[tedster]

Lots of 301 redirects does not cause a true penalty - but it certainly can cause Google to take a long time checking out whether to trust those 301 redirects (black hats have lots of 301 games they play). The times I've heard range from several months to over a year - so massive 301s are not something to undertake lightly!

Depending on the numbers reported in the site: operator can make you crazy. It is not accurate information, as many members here have observed.

One thing you can do is make sure that the xml sitemap only contains the final urls, NOT the ones that are now redirected.

[mcskoufis]

Had a complete CMS/architecture update on an authority site in its niche market (and is not a niche... is a major sector in terms of traffic) and 301s were implemented gradually over a period of 3 months and still having great difficulties.

It is the messiest situation I've personally experienced ever, with several other domains showing the same site...

I really hope this isn't going to last long, but bit by bit, I see a 15% traffic increase week by week, and Google tends to trust our moves...

But not sure what can make this strange filter disappear.. It appears to be gradually increasing the range of keywords sending traffic to the site, with many rankings appearing in the top 10 for an hour or less a day, then sinking down 100+ places.

Think a lot has to do with the October 2009 Google updates [webmasterworld.com].

[chari7ma]

Thank you everyone for your thoughts and insight on my problem. Given me some things to consider and ideas on how to proceed.