Indexed pages drop from 250,000 to 14,000

[giateno]

Hello,

we have a site with free, quality content. We launched it last summer, page indexing and ranking started well and were getting better until the end of september, when the traffic was around 30k unique per month. After that, our site dropped suddenly in rankings, and our traffic was around 0. We made no changes in that period and we do not use heavy SEO practises.

We do monitor the website with google webmaster tools and no problem is pointed out there.

In december the site appeared again in serp and our traffic rose a little, but was about 1/3 of september. We had another drop to zero in february and now we are back to december levels from march.
But from january to march we saw our indexed pages drop a lot, from about 250,000 pages to 14,000. Consider that our site has more than 1 million different pages.

What could we do to have our site not treated well, but at least decently.

Thanks in advance to all of you.

[edited by: tedster at 5:54 pm (utc) on May 2, 2009]

[tedster]

Some kind of ranking and traffic drop a month or two after launching a website is a common experience - but at this point it sounds like something is going on beyond that "honeymoon period" effect. It sounds like the site may not have very strong backlinks/PR - or there may be a technical problem involved, such as too many multiple urls for the same content, duplicate title elements or meta descriptions, server bugs, robots.txt errors - many technical possibilities come to mind.

Have you set up a Google Webmaster Tools account and checked in there for any reports or messages that might be involved in this indexing problem? I'd also suggest an XML sitemap if you aren't already generating one.

[dailypress]

I'd also suggest an XML sitemap if you aren't already generating one.

Tedster, do you update the XML sitemap or do most people have it programmed to do it automatically? Any links where I can learn more?

[tedster]

Google has a nice full Help section about XML sitemaps:
[google.com...]

There are several approaches you can take, depending on your site's size and technical needs.

[giateno]

Thanks for your remarks.

We use webmaster tools and no problems are pointed out there.

Moreover, we have similar sites in other languages, and never had these kind of problems.

Another thing that is strange is googlebot activity graph, shown in webmaster tools. The number of pages scanned per day were high (max 43,000/day) and became low (min 813!) and they do not seem to come back higher. Don't know if there connection with the drop of indexed pages.

[experienced]

we have a site which is having almost 3000 pages with original content. since january 2008 we are trying to get them back in google but google is not indexing them at all. We are regularly adding fresh contnet pages in the site which are required but still there is no improvement. I tried deep search but google did not return enough pages.. although rankings are fine but still i believe if the pages will increase, i can get the more traffic.

its a old and good traffic site...any suggestion.

[peterdaly]

You have 250k pages and your site is less than a year old.

No specifics please, but where did you get the content for these 250k pages?

Is the content unique to the web?

How many of those 250k pages have incoming links from external sources? A better question might be what percentage? 1% of pages with incoming links would be 2500 pages. If only 1% of pages have incoming links, would you index 250k pages deep if you were google? No. (Maybe I'm off base here, but that's my quick thoughts given no context real of the situation.)

I suspect GoogleBot has determined, for either PageRank or quality reasons, that your site is not worthy of the level of crawl depth you would like to have.

To me, even 14k seems like a lot for a >1 year old site.

[SEOPTI]

Peter, I don't think so.

It does not matter how old your site is. Trusted links, PR, good content and not tripping any on-site penalties matters. 250k URLs indexed is not a problem at all for a new site.

[peterdaly]

Trusted links, PR, good content and not tripping any on-site penalties matters.

I don't disagree. I question whether all that is in place, and intended my remarks to dance around those topics.

[johnnie]

If you have a 250k-page site with little inbound links, google might consider your content unworthy. Also, with that many apges, you have to make sure both the spider and any linkjuice can crawl freely throughout your site.

[fishfinger]

Consider that our site has more than 1 million different pages

in just over a year. How many people do you have working on the content for your site?

[jd01]

Unless Google has changed the rules since I have been reading... It has been stated by Matt 'I' Cutts 'The Spam' 5000 pages a day is the max you should add to a site. It seems possible you could have steadily added 2700 (ish) pages a day to get to your current page count, but it also seems possible you tripped the 'too big, too fast' filter by adding too many pages at once...

One thing to keep in mind re this situation is if you 'added steadily' under the threshold, but Google didn't spider all 2700 new URLs one day (or even for a couple of days in a row) and the site is dynamic, which means it probably doesn't serve 'last modified' headers, when G came back and spidered again it could have looked like you added 5400 or 8100 (or more) pages in a single day, even though you really didn't...