hi under the spider spotting chart of searchenginewatch.com i came across three categories of spider normal spider instant spider mega spider how are they classified? what is the difference. does anyone know?
There are instant spiders, but they are currently a joke. (alta's and Lyco's add url spiders - but neither actually add the page). Mega spider is just a Crawler. There are a few left, but they act sporadically and don't index all content. All the current crop of major se's have full crawlers, but they don't index all content. Most often the full crawlers just raid links for link popularity algorythms and words to determine your sites theme - then they throw the page out of the database.