hi under the spider spotting chart of searchenginewatch.com i came across three categories of spider normal spider instant spider mega spider how are they classified? what is the difference. does anyone know?
thanks in advance priya
11:50 am on Mar 7, 2000 (gmt 0)
There are instant spiders, but they are currently a joke. (alta's and Lyco's add url spiders - but neither actually add the page). Mega spider is just a Crawler. There are a few left, but they act sporadically and don't index all content. All the current crop of major se's have full crawlers, but they don't index all content. Most often the full crawlers just raid links for link popularity algorythms and words to determine your sites theme - then they throw the page out of the database.
1:32 pm on Mar 7, 2000 (gmt 0)
"Most often the full crawlers just raid links for link popularity algorythms and words to determine your sites theme - then they throw the page out of the database"
But it's still important to get spidered by these guys for establishing your sites themes isn't it?
3:00 pm on Mar 7, 2000 (gmt 0)
Yes, but you can't neccessarily go looking for that page they spidered in the database. The inktomi/hotbot combo is currently king of throwing out pages while Excite ranks a close second.
11:34 am on Mar 8, 2000 (gmt 0)
thanks for the info by what brett says can i infer that spiders actively index all page contents while crawlers work more for link popularity algo. what are fresh spiders and add url spider