GoogleGuy - 8:51 pm on Jun 6, 2005 (gmt 0)

Whaaa? How did Henk know that I'm really Larry!? Doh, I said it out loud! Doh! Just kidding. :)

I sincerely appreciate it if you've stopped posting documents and taken out the employee's name. Let me tackle the last question you asked:

Please explain. I saw in Eval several duo-list based on the same search terms. Most duo-lists show a different order of answers then the other list. The raters were asked to choose which answers were the best. If this is not filtering, what is it then? I have many other examples.

Think of it like a taste test. If a drink maker had an idea for how to tweak their formula, they might have one version with more vitamin C, or another version with more sugar. It's natural to ask testers for their feedback. But you wouldn't say that the taste testers were the directly changing the formula that was sold in stores. It's even less directly tied with search results: if there's a slight preference for one type of scoring, but it takes 100x the computing power for that ranking, it may well be a better choice to use that 100x computing power for a different task that improves quality more.

I think it's absolutely a great idea to collect feedback/quality ratings about different types of algorithms. But hopefully the analogy of a taste test shows that we may collect feedback without it actively altering our search results.