MEMO TO GOOGLEGUY:
If we take your statement: "If you take as a given that Google wants to give the best page to the user that it can, you're pretty far down that path." as valid, then you are faced with two choices:
1) a significant percentage of the professional webmasters who post here are deluding themselves or lying and they were spamming, had awful sites or were otherwise violating Google rules and have suffered for it;
2) whatever you were trying to fix with this algo went horribly awry and the effect was to dramatically decrease the quality of the "page to the user" favouring spam, 404's and other junk over honest, quality relevant sites.
I realize the sheer scale of Google makes it very difficult to analyze or predict intra-category effects, but you have a pretty strong body of analysis posted on webmasterworld to indicate your algo had unanticipated negative effects on the sites you should value most on your pages.