We've got a network of three large web sites (Around 1 million pages in total). A lot of these pages are generated by a database because of the nature of the site - we sell specific 'widget' parts for specific 'widget' brands (There didn't really seem another feasable way of doing it).
Initially, we had some success and a large percentage of the site was indexed, resulting in some decent traffic. However, just recently nearly all of the pages have become 'supplemental results', and naturally, traffic has taken a hit.
We're trying to establish what the problem is, but not having a great deal of luck. The only potential cause that I can think of is the recent addition of some affiliate links; could this result in being hit so heavily by the search engines?
I think the problem is obvious. Google now requires sites to have a great deal of 'respect' (PR?) in order to index completely, and not have deep pages go supplemental, a 1 Million page site. One way inbound links and site age seem, to me, to be the main criteria in deciding how many levels of pages will be indexed.
I would strongly suggest that a 1 Million page parts site is looking pretty spammy to Google right now... unless you have tons of google 'respect'.