homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
Forum Library, Charter, Moderators: mademetop

General Search Engine Marketing Issues Forum

Independent Relevance Test

 10:18 am on Apr 9, 2003 (gmt 0)

Inktomi outperformed 4 other search engines in a independent search engine test commissioned by Inktomi released last month.


VeriTest also has done reports [veritest.com] commissioned by (among others) Google:




 10:55 am on Apr 9, 2003 (gmt 0)

I'm going to sit down and go through this with a fine toothcomb but my first question I'm asking myself is who were the judges?
When it states that sponsored links were removed does that include any paid for indexing which removes a huge chunk of ink and none of Google.

I'm sure it's mentioned but I couldn't find it on an initial scan through.

Any vaguely scientific report would cover details such as these.

If google commissioned a report which showed it produced the most relevant results, I wouldn't question it. The problem here is that it defies common sense. Therefore it is necessary to tear the research to pieces and see if anything significant stands at the end.


 11:05 am on Apr 9, 2003 (gmt 0)

Inktomi outperformed 4 other search engines in an independent search engine test commissioned by Inktomi.

- HA. Isn't this surfacing about 8 days late?


 11:05 am on Apr 9, 2003 (gmt 0)

Inktomi would only commission the test in the first place if they KNEW they would fair well. Lets be honest its not that hard to do your own relvancy test without having to pay someone to do it for you - unless you wanted the independant publicity.

At the end of the day i am sure if the results were bad Inktomi would not have had them published, then no one would be any the more wiser - to many cynics round here ;)

However its all about how you display the results and the methods you use.

I can do a simple to test to find the best SEO Company in the UK but one of the main rules is that they are based within a 3 mile radius of my office ;)


 11:49 am on Apr 9, 2003 (gmt 0)

It's good to finaly see a secondary power out there.
Doing biz on the net the dependency on a major player is a bit unnerving althoug G eliminates some of the risk by its AD's.

The test looks OK only flawed as based on Ink's basic query log. My experience users on Ink run slightly different q's, bit more adapted to Ink.
Would have loved if they had used WT's all time base or so. Also Ink's base is smaller than G's thus final result relevancy should ultimately be weight by search base.


 11:51 am on Apr 9, 2003 (gmt 0)

What's lacking in the Veritest presentation is a statistical treatment of the results-at least the standard deviation and standard error of the mean. They could have also done something like a t-test (which shows whether the results are too close to call). Since all of these tests are standard, can be found in any spreadsheet program, but are not presented in this report, my guess is that there is no difference between the top two.


 12:22 pm on Apr 9, 2003 (gmt 0)

That they removed queries that did not return at least 10 results on all six engines would slightly bias the results against whichever engine had the largest index would it not? And their results showed Ink & G as very close.


 12:36 pm on Apr 9, 2003 (gmt 0)

There are many searches I monitored over the last year or so, where G and INK were quite close.
In spammy areas both INK and G show messy serps.

The simple fact is Google's algo is not half as good as all the diehard fans religiously believe.
It's a good search engine, and it's true that no other engine is better than Google. They got the largest index, and more fresh results than the competition.
But algo wise ATW and INK and AV are all close or on par.
Everything else is just PR :)


 1:40 pm on Apr 9, 2003 (gmt 0)

It appears the report summarizes a statistical 'dead-heat' with a slight edge to Inktomi.

If the results are that close, it may be a good thing for Yahoo, and put a little less emphasis on many of us that 'live and breath' based on the whims of a single company.



 2:16 pm on Apr 9, 2003 (gmt 0)

>>>There are many searches I monitored over the last year or so, where G and INK were quite close.
In spammy areas both INK and G show messy serps.
The simple fact is Google's algo is not half as good as all the diehard fans religiously believe.
It's a good search engine, and it's true that no other engine is better than Google. They got the largest index, and more fresh results than the competition.
But algo wise ATW and INK and AV are all close or on par.
Everything else is just PR :) <<<<

I 2nd everything Heini says here!


 3:15 pm on Apr 9, 2003 (gmt 0)

Remember that relevancy is only a small part of the value of a search engine. There are several other things that are just as important:
  • Indexing: The spider needs to have indexed a page. If I'm searching for one particular thing on the web and the spider didn't visit the page, then that's a big hit against the search engine. If I go over to the competitor and find the page I was looking for, I may switch permanently.
  • Relevancy of listings: As opposed to the final page visited. The Search Engine Results Page (SERP) must give enough details to make a decision on whether to click through. If I do a search and spend half my time clicking on links and winding up on irrelevant or loosely related spam pages, that engine is worth less than one that has better info in the listings.
  • 404s: How fresh is the index. If all the links on the SERP wind up at 404 pages, I become annoyed and find an engine that spiders more frequently.
  • Cleanness of the UI: How clean is the user interface around the UI. Do I have to watch a bunch of animated GIFs, banner ads, sponsored listings, portal junk, etc? I might use a less relevant engine to avoid all this junk. Inktomi has no control over this, since they don't create user interfaces.


 4:17 pm on Apr 9, 2003 (gmt 0)

While I'm a bit surprised at the results (the closeness, that is), I wonder if the results are compromised via usage of only Inktomi's query log (as an initial start). The test would be more meaningful for me- if they either used something like the Lycos 50 interspersed with longer keyword phrases (say lifted from the headlines or top blogs) or if they had requested query logs from each of the engines being considered- and used a random sample of those queries (after their cleaning process). Also, if they had said anything about how they selected their judges, that would have been helpful as well... e.g. random? people off the street? experienced searchers?

I'm glad they provided the keywords because it enables any of us here to do random testing. Admittedly I only did five searches (picked at random from their list). However, for how I understand "relevance", Google has the slight edge not simply with regard to the content but also with the presentation of results. This can be important although this factor was eliminated from the test and from being presented to the judges- since they only saw the queries and the list of links- and NOT how the results are displayed. Google results tend to highlight and repeat the keyword phrases in a way that's easier for me to determine if I even want to visit the page to determine relevance. But this could be a subjective preference.


 5:23 pm on Apr 9, 2003 (gmt 0)

Interesting.. Correct me if I'm wrong, but VeriTest and eTesting Labs are the same folks, right?

Coincidentally, when AltaVista commissioned eTesting Labs to compare search engines, AltaVista was judged the most relevant:

Strangely enough, when Ask Jeeves commissioned eTesting Labs to compare self-service search interfaces, the AskJeeves product came out on top:

It helps a lot to pick the ground rules, what queries to throw out, etc. :)

jeremy goodrich

 5:27 pm on Apr 9, 2003 (gmt 0)

Nice, GoogleGuy, that is kind of funny how it seems to favor the person that sponsors the study.

Perhaps if I commission a study for my own site, it'll beat Google, too?

Somehow, I don't think a simple, "study" will get people to switch to 'brand x' from Google.

Even with a bias in the study, imho, the results from Inktomi are 2nd only to Google.

Now, if they would only kick out all the paid for spam (XML CPC stuff) then they'd have quite an engine.

Not to mention that I'm sure the FTC will do something, at some point, to all the non labeled advertising being floated around...

Mikkel Svendsen

 7:07 pm on Apr 9, 2003 (gmt 0)

This not really news - at least not to people that have been testing relevancy for a long time. I work for some large portals in Europe that have been monitoring relevancy for a long time to be sure they have the best search partner possible. These results have never been public as it is only for the portals use, so they gain nothing from them - except for making a better portal. Still, they come up with very similar results. Google is not allways the most relevant search engine. Inktomi and FAST are some times better - and some times not.

If you look at relevancy historically Inktomi and FAST have been doing much better than Google. Google is a good search engine - they just haven't improved much over the last year or so. FAST and Inktomi have.

Anyway, this study show the need for a standard for relevancy, as spelled out by Danny Sullivan some weeks ago in ClickZ. If the search engines do not work together on a standard others will - and they most likely won't like it.

Whenever search engines come out with a new version they brag about:

1. How fresh they are - and they have numbers to prove it

2. How big they are - and they have numbers to prove it

3. How relevant they are - but they have NO numbers to prove it

So, Dear Search Engines, get your act together and make that standard. Also, remember to show all results - both good and bad and not like we have seen it with the NPD-studies where only the good results are made public.


 9:32 pm on Apr 9, 2003 (gmt 0)

damn, talk about the ultimate "yes man" LOL,
I guess you get what you pay for...literally


 2:24 pm on Apr 10, 2003 (gmt 0)

Interesting, and good post, Mikkel.
Now that would be fantastic: an objective, third party relvancy test, held perhaps twice a year, augmented by an international board.
Would give the SEs quite an incentive to work on their algos, especially for new and second tier engines. I'm sure it would lead to great improvement across the board.

As to the results of the portal test you mention: that comes as no real surprise to any open minded user.
Google is vastly overrated, even by experienced webusers.
I think it's true that Google has not substantially improved their relevancy for a long time. They are still good though. The question is just how scalable their algo is.


 12:48 am on Apr 14, 2003 (gmt 0)

My biggest question here is - So what? Is there a search engine ANYWHERE that Uses Inktomi as its results on the first page? The simple fact is, Google's results ALWAYS appear on page one of any Portal it partners with, as well as its own site.

So the question of how relevant Inktomi results are would only be useful in convincing Yahoo! to ditch Google for its search results. For the average user, it is (next to) useless!


 4:11 pm on Apr 17, 2003 (gmt 0)

Who do you think Yahoo bought recently?

Hint: It's not Google!


 7:01 pm on Apr 17, 2003 (gmt 0)

I think a relevancy test as proposed by Mikel has to be restricted to just test the relevancy of the results - and nothing else: no layout, no speed and so on.

Only this way you can present the result urls in an SE-independent style, so that the jugdes make their unbiased decisions.

And of course you'd need a wide variation of judges, to represent all internet users. Difficult. But it would be worth the effort.


 10:01 pm on Apr 17, 2003 (gmt 0)

Hmmmm. A test run by inktomi, with inktomi employees, employee picked judges...and the winner is Tada Inktomi. I read over the actual test it seemed to be interesting enough. But the main problem with it was the rating and the keywords. The key words that were used could have been easly picked by anyone, but the fact of the matter is that Inktomi picked them which could be very skewed(sp?). As far as the keyword selection Inktomi should have conducted a study as to the most popular searches, excluding adult material, and then conducted the test. That way your not getting "Billy Bobs 1967 Ford Pickup Truck Pictures." There were one or two that left me scratching my head.

There is another industry that is a hobby of mine and there are several magizines to support it. But there is one that is independant and gives unaltered results and for that reason I trust it more than the others.

When you have your own search engine or directory it is easy to point out all of your good points and not the bad. Which is what I am seeing here. Another downside to this study is that most of these are pay for click/preformance. Google is not and the rest I think are paid, I think. So automatically you will have skewed results. IMHO this points out that even at Inktomis best they are still below googles index. Because Google is free and Inktomi charges for listings as do the rest.

There is an old saying one of my Gunnys told me "You cant polish a turd." I think that it fits in this instance. No matter how hard inktomi tries they will still be sub par with google as long as they are providing paid search results to googles free.

However the good side of this bias test is that Inktomi can "say" that they provide the best search results, but just barely.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Marketing and Biz Dev / General Search Engine Marketing Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved