site:mydomain shows 6,060 results - but only 349 actually showing

Googlebot crawling, finding pages, adding to index but not displaying them

[fhugh]

Hi guys,

Our company recently took over a website that has been online in various shapes and forms since 1998. The previous developer made a complete mess of the previous site, using frames and other silly things which resulted in the site dropping out of the rankings.

In January we re-launched the site using a far better content management system we developed in-house. Googlebot came a crawling, and within two weeks our site was #2 for our main required keyphrase. Google had also indexed our top level categories and seemed to be deep crawling the site.

Then, amazingly, Google dropped the new top level category pages and left only two top level category pages in there. When I do a site:mydomain.com Google says 'Results 1 - 10 of about 6,210 from mydomain.com'. However, when I browse the results that are showing in the results for site:mydomain, I get as far as 'Results 341 - 349 of about 6,210 from mydomain.com', and then it runs out of pages to show!

Clearly Google has crawled the new content, and knows that there are 6000 odd new pages ready to be shown in the results, but its not displaying them in the search results.

Any ideas as to whats going on?

H

[jonrichd]

I am going to assume that you've clicked through any 'Repeat the search with omitted results' message that you've gotten.

I've seen situations like this with search terms that are relatively broad, with 6 million plus results. After about 350, Google gives up, and won't show anything more.

I don't have an explanation for this, other than Google can't find any more pages for the broad term that do a better or different job of describing the subject.

So, I would ask are your pages so similar that Google might feel that there's nothing gained by displaying more results.

Another tack you could try would be searching for a phrase that's on one of the pages that Google doesn't list, but has crawled. If the page shows up in the serps, then it's indexed, but G doesn't show it in a site: search.

[texasville]

Your story sounds very familiar and is resonating throughout the forum. Start reading here: [webmasterworld.com...] and you will find my theory. It may or may not hold water as we say down here.

[fhugh]

Interesting theories... its wierd though, that the top level channel pages came up, then bigdaddy occured, and now no new pages are appearing in the index, and the initial pages that had been added have been dropped.

Odd. Googleguy?

[g1smd]

Often it is a duplicate content problem.

If you have both www and non-www pages listed then get the 301 redirect installed immediately. If you have several domains, perhaps .com and .org and .co.uk, etc, then make sure you 301 redirect to the one that you want to be listed.

It is more likely that the title and meta description are not unique per page, and Google filters the extra pages out. Make sure that every page has a unique title and meta description and that they describe exactly what is on the page that they are placed on.

Also make sure that each page has only one URL that can reach it. Use Xenu LinkSleuth to check your site out.

[fhugh]

Heya,

No, its actually one domain, all unique content too, and google has been told to look at www.mydomain.com stuff, and not mydomain.com stuff. Seems to be even more pages 'in the index' today, but just not showing!

Getting frustrated now - Google better have a good reason for this.

[trinorthlighting]

Umm, look at the thread about google not having enough storage space on their servers..

[g1smd]

Disk Full: Abort, Retry, Fail?