Listing a big site

Partially crawled; listed; then delisted; why?

[simonv]

We have a site with 3.2 million static pages - all different.
Immediately pre the March Google Dance we had about 400,000 pages crawled and in the April stats, increased our listings from 250,000 pages to 500,000 pages.
We were little crawled in March. None of our pages changed significantly.
In the April Dance, we lost 300,000 pages and dropped to 200,000 listed. I cannot find out why. All pages were different and not spam.
Also I have tried for years to find a way to list most of our 3.2 million pages but Google Sales seem not to have any kind of plan to permit this to happen.
Any ideas?

[edited by: heini at 12:33 am (utc) on April 24, 2003]
[edit reason] Yep, no urls per TOS please / thanks! [/edit]

[vincevincevince]

did you make a sitemap, listing all the pages from one long list?

and please remove the specifics ;) ie the url

[simonv]

No, but we do have several thousand index pages which, taken together, would point to every detail page (and there is a Previous Page/Next Page link and an index page link on every individual product page).
Do you think a huge sitemap index page would ever be seriously crawled?

[Oaf357]

You might want to go with a simplified site map of sorts. Basiscally listing sections and sub-sections. It would help. 3.2 million pages though, that's an awful lot for Google to suck down, maintain, and freshbot in their database.

[simonv]

Well I naively believed that if Amazon can get 4.5 million pages listed and B&N about 1.2 million, we should be able to do better than 200,000! Thanks for the advice.

[simonv]

BTW, forgot to mention that the Froogle guys say they will accept a feed of all 3 mill, but they don't know when Froogle will be able to accept that many (while in Beta they only wanted 100,000 pages).