Xoc's Search Engine Rating system

[Xoc]

There are three factors in determining the quality of a spidering search engine (SE).

SQ (spider quality): The spider must come by frequently and must crawl deep. Must honor robots.txt. The results of spidering must then be rolled into online database quickly.

SR (search engine results page): The SERP must then show relevant results based off the data they have collected from the spider. Summary must give info on whether to click through or not. Page must be pleasant. 404s must be few and far between.

CS (customer service): Response from a human to questions in a quick and reasonable manner. Free and easy submit pages.

Let's call my ranking system the Xoc Scale (XS). It awards an engine up to 5 points for SQ, and 5 more for SR. Now you could also give points for Customer Service (CS), but the End-User (EU) doesn't care much about that, and the EU is picking the SE. So a CS score is placed in parens, but doesn't count toward XS ranking.

EUs are slow to change SEs, but if they continue to get bad results, they eventually decide to change, and they try a few engines, listen to friends, etc. So I believe the Xoc Scale eventually determines which engine they will likely switch to.

Based off this, and my own subjective analysis of my log files, I rate them like this:

1) Google: Spider is excellent, crawls deep. The database takes four weeks to update, which could be better. 4.5SQ. SERP is relevant. Usually the page I'm looking for is in the top few listings, and rarely after position 20. Listings could be a little clearer and allow better information to decide if I want to click-through. 4.5SR. Customer Service seems good and responsive in the one contact I've had with them, although the final outcome of what the engineers did could be slightly better; still they did something to their algorithm based on my request within 4 weeks. 4.5CS. FINAL RANKING: 9.0XS (4.5CS)

2) Fast: Spider is excellent, crawls deep. In my experience it crawls non-PDF files and dynamic content deeper and more frequently than Google. All pages that it finds get indexed. Needs PDF and dynamic content results. 4.5SQ. SR, though, needs a lot of work. The top results are just not relevant and pages that I'm looking for are frequently buried deep. 3.0SR. Submit page is fine, have had no dealings with customer service. Not enough info to rate CS. FINAL RANKING: 7.5XS(?CS)

3) AltaVista: Spider is good when you can get it, but not as good as Google or Fast. Results are slow to make their way into the database. 3.0SQ. Result pages are easy to read, but relevancy has declined a huge amount. 3.0SR. Customer Service got back to me in a few hours on my one contact with them. Submit page works well. 4.5CS (although others may feel different). FINAL RANKING: 6.0XS(4.5CS)

4) Inktomi: Spider is slow. Usually hits my home page once every couple of days, then goes away. Results are even slower to get online. 2SQ. Results page are relatively easy to read, and relevancy seems good, although outdated. 3.5SR. Have had no contact with CS. FINAL RANKING: 5.5XS(?CS)

5) Northern Light: Spider comes by frequently. Crawls moderately deep. Don't have enough info on how frequently it updates the database. 3.0?SQ. I find the SERP hard on the eyes and relevancy is low. 2.5SR No contact with CS. FINAL RANKING: 5.5?XS(?CS)

6) Excite: Spider is slow. Usually hits my home page once every couple of days, then goes away. Results are even slower to get online. 2SQ. Results page allow a lot of spam, and because of slow updates have a lot of 404s. 2.5SR. No contact with CS. 2SQ. FINAL RANKING: 4.5XS(?CS)

Now if Excite is changing their ways with this recent update, I'll rank them higher. They do need a better spam filter.

What do you think?

[Brett_Tabke]

That is such a good system. The one variable you haven't accounted for -- and the most important to most Wmw members -- is Referrals, and quality of referrals. The proof is in the pudding as they say.

[Xoc]

True, I didn't count for referrals. On the other hand, I think ultimately, the Xoc Scale will exactly turn into the proportion of referrals, since I believe that end-users will switch to the various engines in proportion to the rating. There may be a time-lag of a year, though.

[mivox]

Quality of referrals per SE would vary widely depending on your target audience... it would be really hard to do an overall average for referral quality.

Tacky knick-knacks and cute baubles for sale? Referral quality would probably be high from AOL, WebTV and national ISP default SEs (MSN?).

Aiming for a techie audience? Quality of referrals would be higher from Google and some of the 'obscure' geeky/old-school sites.

[Xoc]

BTW, I should note that I downgraded Northern Light's SR to 1.5. The relevancy of the top ranked pages is just terrible. So NL should be FINAL RANKING: 4.5?XS(?CS)

[Xoc]

One year later...what has happened? How good is the rating system?

Google is now the dominant search engine. Fast is not well-known in the U.S., but in Europe is hot. AltaVista is essentially dead, although technically still alive. Inktomi as a search engine is dead, too, but figured out how to make money anyway with pay for indexing. Northern Light is dead. Excite is dead.

I think the system is an almost perfect predictor of what happened.

[Brad]

Xoc,

I think it is an excellent system. I would like to see if you think there have been any changes on the ratings of those SE's left standing.

[mbauser2]

Here's mbauser2's search engine rating system:

Whichever engines SEO's complain about the most must be the best.

The winners, by a mile: Google and ODP.

But then again, I'm a old cynic. Feel free to ignore me.