Genuine WebCrawlers

rather than Spider on Submission

[IanTurner]

Having looked at this topic, it seems that some of the search engines that claim to spider your site only do so if you submit your site to them and that very few engines actually have true webcrawlers which index pages automatically from a starting point.

I'm looking to identify the true webcrawlers (associated with Search Engine) - my list at the moment is as follows:

Bigger Players
--------------
Slurp - Inktomi
Googlebot - Google
FAST WebCrawler - FAST
Scooter - Altavista
Openfind - Openfind
(Teomagent - Teoma)
(Zyborg - Wisenut)

Smaller Local Engines
---------------------
Henry - Mirago
Szukacz - Szukacz
KIT - Fireball

Those in brackets I haven't seen for a while.

As far as I can tell other spiders don't actual crawl the web and I'm not 100% sure of Zyborg at the moment.

Can anyone else add any other genuine search engine crawlers. (I don't mean ia_archiver or turnitin bot which don't power search engines)

[caine]

Ian,

I think you have pretty much encapsulated the Crawler market. The only one's that i can see being outside of the ones mentioned that maybe genuine crawlers, are the Student-Business types, though there may be others in Asia that are of reasonable size in distant places.