Google crawling heavily but search results are disappearing

[gibbs07]

Hello
I am having a nightmare and need to share this story.

I have a site which I managed to get over 200,000 urls cached.

The problem is this caching has been going on for 3 months since
the beginning of the year.

Google has been performing GETS at the rate of 1 per second on some days
and on other days typically 5 gets per minute.

What is abd that during this time the search results disappear for weeks
and the suddenly come back only to disappear again

When the results do appear only a fraction of them are there, they last for a week and then disappear again.

I used a sitemap which had all the 200,000 plus urls which was
great but it seems like Google will not stop.

I need my site back . I am happy that all these internal links are there but it is driving me crazy and possibly out of business.

Has anyone vere experienced something like this?

[tedster]

Hello gibbs07, and welcome to the forums.

The first thing I'd check is whether there are some problematic server responses - especially 404 handling that doesn't return a true 404 http header, or dynamic urls constructed in such a way that there is more than one address for the same content.

If you don't already have a Webmaster Tools account, then set one up to see the feedback Google can give you about their crawl of the site.

[gibbs07]

Hi Tedster

I have no server problems as regards 404 . I do have a webmaster account.
What could help is if I could contact someone from Google and
tell them my problem but there seems no way of doing this directly,hence
why I am using this forum. Basically I need to speak to someone with a large databse driven site

[whoisgregg]

Is the main problem that Google is crawling too fast? If so, you can go into your Webmaster Tools account and limit the crawl rate of googlebot.

Or is the main problem that your rankings appear and disappear? If so, that will be a much more complex issue... What have you changed about your site recently?

[gibbs07]

Hi
The problem is not te crawling but the serach results continually
disapppearing then suddenly re-appearing and then disappearing again
My site is quite large which explains the crawling. My site had 8,000
pages cached in December 2007 it has now jumped to 215,000 pages cached which is great but the search results are unstable. There are only 215,000 pages on my site which is waht i submitted via the site map.
Google seems to want to continually fetch over and over again..why?

It is putting me out of business

[soxos]

These extra 200k pages, where did they come from -
From a content point of view, are they new content, or re-distributed content used to create new pages? I just want to rule out dup content. Especially if all these pages have been added to your site at once.

[gibbs07]

hi
the content is from affiliate product lines all original

[soxos]

"affiliate product lines all original "

Isn't this a contradiction? if its affiliate content then are you sure it is original? It would be unlikely...

Why don't you try searching for a snippet of text on one of the pages to see who else has the same content.

I think it is probably these pages causing the problem, as Google might be seeing them as duplicate content. Especially if they have been added all at once.

Firstly check out if / how often the content is being used by other people, if it turns out the content is widely used....