types of spiders

[priya]

hi
under the spider spotting chart of searchenginewatch.com i came across three categories of spider
normal spider
instant spider
mega spider
how are they classified? what is the difference. does anyone know?

thanks in advance
priya

[Brett_Tabke]

There are instant spiders, but they are currently a joke. (alta's and Lyco's add url spiders - but neither actually add the page). Mega spider is just a Crawler. There are a few left, but they act sporadically and don't index all content. All the current crop of major se's have full crawlers, but they don't index all content. Most often the full crawlers just raid links for link popularity algorythms and words to determine your sites theme - then they throw the page out of the database.

[Determined]

"Most often the full crawlers just raid links for link popularity algorythms and words to determine your sites theme - then they throw the page out of the database"

But it's still important to get spidered by these guys for establishing your sites themes isn't it?

[Brett_Tabke]

Yes, but you can't neccessarily go looking for that page they spidered in the database. The inktomi/hotbot combo is currently king of throwing out pages while Excite ranks a close second.

[priya]

thanks for the info
by what brett says can i infer that spiders actively index all page contents while crawlers work more for link popularity algo.
what are fresh spiders and add url spider

priya