Lapizuli - 5:34 pm on Jul 9, 2011 (gmt 0)

You leaving a website is not something we see - in fact, if you left the website, we'd assume you were more satified than less satisfied simply because you didn't come back to us.

Scarily false assumption that might mask users' dissatisfaction with the search engine, itself. Users who are unsatisfied with search results from a particular search engine and have exhausted their attempts with one may simply go from the last website to the next SE by typing it in the URL.

It's that quick, immediate "back" action that we're interested in. Yes, you could argue in isolated instances this is not a realistic metric for a specific exmaple, but across the volumes of queries we see, it's valid.

I see what you're saying, but unless you're using a richer approach to interpreting the meanings of those patterns in those millions of queries, their actual number means little. A whole bunch of people may do the same thing after doing a particular search, but without a really microcosmic focus on that search, you'll never know if you're interpreting it correctly, or if you're simply being consistently wrong in such a way that users have learned to effectively mask their dissatisfaction by adapting their search skills or behaviors - which happens either when a less-than-optimal-but-marginally-acceptable result is available or when no better search engine is available (and no offense, but these days they are all pretty equal.)

Anecdotally, I'd say none of the search engines are very good at reading the minds of users in long tail queries (which is certainly the case in our household - a typical web hunt requires five or more successive searches these days, quite often going from more specific to less specific, a reversal of the way it was a few years ago, when users' search skills were rougher than the search engines' brains. )

They're fine at reading queries with shopping intentions or quick reference intentions, and improving for navigational searches, but for informational intentions, the results are really, really rough...even for queries that aren't those desperately long, unique ones, but obviously oft-searched because they're echoed in the suggested searches under the search bar.

What I'm saying is, if volume was the only issue, if you could do it for 10,000 or 1,000 searches, you should be able to do it to a lesser extent, but still on the right track, for 100 searches and even 10 - if your analytical voodoo were really putting you in touch with the minds of your users.