BigDave - 9:09 pm on Feb 17, 2003 (gmt 0)

My big problem with the other engines is that they just do not seem to take their spidering seriously.

I agree that several of them have much nicer features for doing advanced searches. I don't generally need help coming up with search terms that work, so that really isn't and issue for me, but I imagine it can be for others.

As a webmaster, I am greatly biased towards Google, simply because they are the only ones that seem to condsider my site worth fully indexing. I was completely indexed by the third month with around 2000 pages in google.

Now we are at 6 months, and these are the number of bot visits I have had so far this month

Googlebot (Google) 3616
Inktomi Slurp 1444
Road Runner: The ImageScape Robot 523
Scooter (AltaVista) 344
Fast-Webcrawler (AllTheWeb) 86
Alexa (IA Archiver) 69

Ink discovered my site and started going nuts about 2 days after yahoo bought them. Before that they had 2 of my pages.

AllTheWeb has 54 pages, and so far it looks like they only added 2 more to the pages that they crawled this month. ATW simply does not live up to their name.

I ran some tests between ATW and google on several of the sites that I frequent, that are all content rich and major players in their industry. On sites with less than 100 pages, ATW generally had slightly over half as many pages as google did for the site. On sites that have hundreds, or even thousands of pages, ATW rarely had over 100 of the pages indexed. The only time that they seemed to have much more than that was for the true king of the hill sites like nationalgeographic.com.

It doesn't matter how nice their UI is, if they don't have the content to search.

And as a webmaster, I do take it somewhat personally that they don't seem to consider my site worth deep crawling. I will continue to favor the SE that favors me.