steveb - 9:55 pm on Nov 12, 2006 (gmt 0)

In short, the problem is the new Googlebot is so weak that while it may crawl a main page at a site, it often does not drill deeper on sites with low PR and/or few links. It's like Slurp used to be a lot of the time, banging its head against the index page of a domain (and robots.txt) and then skipping off like a stone across a lake.

IF a domain is healthy (and the vast majority of supplements come from non-health) then more links to your homepage, more link paths to internal pages, more PR, more more more... this is what you need to drive Googlebot deeper into your domain to grab current versions of your pages to put in the main index. The supplementals will still be there in the background, waiting to appear if your indexed pages are dropped from the main index again.