Typical Percentage of Indexed URLs to Total URLs

[chrisnewson]

What ratios do people tend to see in Google for indexed URLs to Total URLs.

We have a PR5 largely forum based site that's been in archive.org and growing steadily since September 2004.

We have about 700,000 of 11 million pages indexed. We use google sitemaps and webmaster tools confirms it has 11 million Total URLs, but only indexed 700,000 - 6%.

Our other main site is not a forum, has PR5 and 140,000 Total URLs, with 135,000 Indexed - 96%.

I don't know whether I need to act on the 6%, or whether I should expect that for such a large site of that type and standing. What do other people find?

[tedster]

With forum sites, a low percentage indexed is common - although 6% sounds very low. With a technically sound but non-user content site, I've seen over 90% of urls indexed. That's possible even on sites with around 1 million urls.

You might want to check out this thread, from the time that Google rolled out their new Big Daddy infrastructure, for ideas about any way in which your forum's urls might be a problem for Google -- especially duplicate urls for the same content:

Huge decrease in number of forum pages indexed [webmasterworld.com]

[chrisnewson]

Thanks tedster. Much appreciated.

*scurries off to look at that URL*

[jimbeetle]

I thought I replied to this the other day. I apparently did my usual preview, looks good, forgot to hit submit and backed out to recent posts.

What Ted said above, plus can you clarify this:

We have about 700,000 of 11 million pages indexed. We use google sitemaps and webmaster tools confirms it has 11 million Total URLs, but only indexed 700,000 - 6%.

There can be quite a difference between "pages" and "URLs", especially with forum software which can generate multiple URLs for each "page" of content. Does the forum actually have 11 million unique pages or a smaller number reachable several different ways?

[chrisnewson]

We've had about 9 million posts which appear on individual post pages.

Posts also appear on thread pages with between 1 and 20 other posts. We have 500,000 ish thread pages.

We also have 300ish forums, 200,000 member profiles, 10,000 wiki pages etc..

So all in all in our sitemap files we've about 11 million unique pages - on 11 million unique URLS (albeit a post will be duplicate content of 1 part of a thread page)

Having read the thread tedster recommended I understand that when we are spidered we will be presenting a lot more than 11 million URLs, but we are only submitting 11 million unique ones in the Google sitemap.

We use vbulletin.

[chrisnewson]

Does this shed any more light on things / ring any alarm bells?

site:mysite.co.uk *** -sljktf = 2.7 Million pages

site:mysite.co.uk = 746,000 pages

single post site:mysite.co.uk 2.6 Million pages

It's obviously not the full 11 million in any case, but it seems like a large number of pages are there but not in the main index.

[Asia_Expat]

Over the last week or so, I noticed some movement in the amount of pages I have in the main index. It's been at around 350 forever it would seem (supp varies around 4500 - 5200, mostly well optimised forum content).

I noticed some positive movement in my Google traffric so I started closely watching supp numbers and sure enough the pages in the main index are on the rise. It's not a sudden rise and they've slowly climbed to today's 620 in the main index. Number of total pages including the supplemental index remains unchanged.

This is mirrored in my Google traffic which has remained at around 350 per day forever, now standing at around 500 per day. I don't know if this trend has levelled out yet (I sure hope it continues).

To me, it's perfectly clear that any page in the supplemental index is nowhere.

[edited by: Asia_Expat at 5:47 am (utc) on Mar. 17, 2008]