Forum Moderators: open
For the press release, see: [psu.edu...]
For those interested in this type of analysis, is this the type of information that a Web site developer or online organization can use? If not, what would be helpful?
I can send you the research paper if you want.
Jim Jansen
How does this differ from the information that has already been gathered and presented regarding these issues?
Also, why do you think this would significantly alter web surfing?
"Niche search engines that focus on a narrow topic or search engines that cluster results by finding similarities and grouping them may be consumers' best bet for improving relevancy."
Finally, did the study give any indication of surfer's satisfaction with results?
"About 55 percent of users checked out one result only" - Couldn't this indicate the surfers are satisifed with their results?
We have a thread titled "Ugly Sites Sell", and this has been a discussion we've had for some time.
You mentioned:
“People make instantaneous judgments about whether to stay on a site, and if a site doesn't the give the right impression, users will bypass it,"
I've never been able to understand why sites that aren't very well designed are the ones that convert best. In my opinion, those sites wouldn't make the right impression, but they seem to because people buy.
Any ideas on how to qualify more precisely what "the right impression" means?
Studies with stats are hard to come by these days. Thanks for the link. That paper sounds interesting indeed.
I pulled this from your page:
>>relevance was determined
I was wondering how you determined "relevance". Did you use simple use keyword matches or did the searchers disclose the information they were seeking? The research we've done indicates that people aren't particularly adept at phrasing their queries so while the relevance of the search term used to the results displayed is quite good the searcher is still frustrated. We've also found that the dissatisfaction increases for people that use English as a second language.
Did you group multi-word queries and determine relevance among those respective groups? Were single word queries tossed out? Misspelled words?
[edited by: digitalghost at 8:46 pm (utc) on June 30, 2003]
I'd suggest
"A page has to give the right impression (i.e., project authority on the subject), easy to load and relevant to a searcher's needs."
1. I was wondering how you determined "relevance
>> We had three independent evaluators view and rate 530 Web pages that search engine users viewed. Evaluation was based on the complete query that the actual user entered. In information science, this is called topical relevance. Interater reliability was really high at 0.95.
2. Any ideas on how to qualify more precisely what "the right impression" means?
>> No I don't. But, there appears to be an overall view that seachers make the decision in seconds. The "right impression is probably a complex mix of relevance, HCI, network load time, and user needs.
3. How does this differ from the information that has already been gathered and presented regarding these issues?
>> First study using real users, real queries, with click thru data.
4. Finally, did the study give any indication of surfer's satisfaction with results?
>> We did a relevance check (see 1 above). Precision was 52%.
5. Also, why do you think this would significantly alter web surfing?
>> After ranking of results, which almost all search engines do now, I think clustering is the next logical step.
6. "About 55 percent of users checked out one result only" - Couldn't this indicate the surfers are satisifed with their results?
>> Yes, see #1 and #4 above.
Vannevar Bush is screaming for the protocol to be published. ;)
A quote from Mildrid Ljosland:
Researchers like Su [14, 15] Green [7] and Borland and Ingwersen [2] argue that relevance is a complex item with many topical aspects, one of them being who is assigning the relevance, another being which relevance is considered.
The subjectivity of relevance would suggest that three independent evaluators could assign relevance values, while a user performing a given query in which relevance has been determined by the evaluators, finds the results of the search to be less than satisfactory.
In other words, if relevance was assigned by the evaluators and the surfers weren't questioned as to their satisfaction with specific results then surfer satisfaction was also assigned by the evaluators. Was the query/result match the sole basis for determining relevance?
we have already been discussing that study of yours, commented on by the BBC, but could not find the orginal paper on-line:
[webmasterworld.com...]
On impatient searchers and Alltheweb results.
This thread also covers impatience according to Jakob: [webmasterworld.com...]
Any data on what percentage of the 450.000 Alltheweb queries were disregarded completely?
(that is no one clicked on any of the results - e.g. rank checking stuff)
>> Sorry, on vacation at the moment. When I return, I publish the paper on my Web site, and post the URL.
Jim
>> You right in that topical relevance does not capture the complex richness of relevance. However, with the inability to collect survey data from the users, three independent evaluators is a good "reasonable person" approach to relevance.
Besides, survey data also have disadvantages. As do, lab studies, and transaction log analysis, etc. One just has be to aware of the limitations.
Thanks for reference to Green's work.
Jim
[ist.psu.edu...]
Jim
