Edwin - 12:31 pm on Dec 28, 2006 (gmt 0)

Here's something to think about (and I build content sites too, so I realise that my own foot is one of the ones that I might potentially be lining up to shoot with this)... how often in 2006 (and soon to be 2007) does a significant new resource come along about any given niche topic that is really head and shoulders above the existing corpus of material on that topic?

For example, if we're talking about widgets, and there are 50,000 pages that relate directly to widgets, how often will a new site/page be created that is REALLY hand-on-heart, all-biases-aside genuinely better and more useful than say the top 100 biggest/most comprehensive/most popular/most in-depth of the 50,000 pages that have come before it?

Seems to me that there needs to be a way to establish a baseline "list of useful sites/pages" about any given niche topic, and then a separate and complementary process to elevate the relatively rare new-but-worthy site/page to be included in that list.

After all, as was already brought up in this thread, the search engines only show the first 1,000 results for a query in practice, regardless of how many results exist - and of those 1,000 results only the first few pages' worth are of any real consequence unless we want to start delving into the fractions of 1% of overall traffic that hit the deeper pages within those 1,000 results.

Here's the billion dollar question: Does a search engine USER get a better experience from seeing 78,000,000 results for a given query, or 100 (or 200 or 500 or whatever) hand-picked, relevant results, with zero spam or off-topic material?

Take a really deep breath, and step away from the webmaster aspects for a minute and say to yourself, which of these experiences would leave an average user happier, more informed, more likely to use that particular search engine again?

Does seeing "78,000,000 results" say to the average user "we're incredibly thorough, and you won't miss ANY information on this topic, no matter how insignificant" or does it say "good luck finding the needle of information you really need in this massive haystack of information we can offer you"?

One way to improve the perceived search experience could be to use existing search popularity data to help focus the manual fine-tuning effort, and gradually move down the long tail as the "big" subjects get taken care of.

First, you fine-tune the results for the top 10,000 searches (and synonyms thereof), then the next 100,000 and so on down the line. The smaller the number of searches, the less important it is to fine-tune them initially since the vast majority of people won't ever see them.

Even when people hit on "untuned" results - which would happen relatively often at first - they would also hit on tuned results even more often... so the overall impression of this hypothetical new search player might still be better than that formed from existing competitors which rely primarily (or solely) on smart algorithms.

Even if you lined up every "webmaster" in the world, plus everyone who's ever started a blog (and kept it going long enough to "care" that it got indexed by the search engines), or posted a list of URLs on a social bookmarking site, that's just a fraction of the total online population. Consumers of information outnumber regular producers of information by (probably - I don't have the exact stats) 50:1 and they outnumber commercial producers of information (those who depend on the web for income) by perhaps 200:1 or more.

So if you can come up with a way to satisfy those average users, you could theoretically be onto a winner even if you end up stepping on the toes of 95% of all webmasters out there (by snubbing their sites) in doing so.

[edited by: Edwin at 12:35 pm (utc) on Dec. 28, 2006]