If you have multiple URLs leading to the same content, "duplicate content", some of the URLs will appear as normal results and some will appear as Supplemental Results. The Supplemental Results will hang around for a long time, even if the page is edited or is deleted. Google might filter out some of the duplicates, removing them from their index: in that case what is left might just be a URL that is a Supplemental Result.
The fix for this is to make sure that every page has only one URL that can access it; make sure that any alternatives cannot be indexed. Run Xenu LinkSleuth over the site and make sure that you fix every problem found. Additionally do make sure that you have a site-wide 301 redirect from non-www to www as that is another form of duplicate content waiting to cause you trouble.
Also, make sure that every page has a unique title tag and a unique meta description, as failing to do so is another problem that can hurt a site.
Many thanks for that thorough reply, a nice summary of possible problems.
What do you suggest if none of these issues apply. All I can think of is when we launched we weren't aware of the www / non www problem but then put this right with a 301 redirect months ago.
I may be over-simplifying this whole issue, but this whole supplemental thing seems very over complicated and no-suprising that google's indexing is in such a mess.
Why not:- index everything, and rank what it deems more relevant over the non relevant for the search phrase entered? It seems a massive waste of resources to cache different version of the pages then revert back to old versions.
Let's say that google deems a page to be dupe enough to make it supplemental (wrongly in some cases) and therefore it punishes the content of that page and doesn't show any very specific search phrases as results. What if that page was the only page on the web that contained information on that search term?
The user isn't getting the best result possible.
<edit reason: format quote>
[edited by: tedster at 7:12 pm (utc) on Aug. 17, 2006]