Welcome to WebmasterWorld Guest from 54.227.110.209

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

types of spiders

     

priya

11:28 am on Mar 7, 2000 (gmt 0)

10+ Year Member



hi
under the spider spotting chart of searchenginewatch.com i came across three categories of spider
normal spider
instant spider
mega spider
how are they classified? what is the difference. does anyone know?

thanks in advance
priya

Brett_Tabke

11:50 am on Mar 7, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



There are instant spiders, but they are currently a joke. (alta's and Lyco's add url spiders - but neither actually add the page). Mega spider is just a Crawler. There are a few left, but they act sporadically and don't index all content. All the current crop of major se's have full crawlers, but they don't index all content. Most often the full crawlers just raid links for link popularity algorythms and words to determine your sites theme - then they throw the page out of the database.

Determined

1:32 pm on Mar 7, 2000 (gmt 0)

10+ Year Member



"Most often the full crawlers just raid links for link popularity algorythms and words to determine your sites theme - then they throw the page out of the database"

But it's still important to get spidered by these guys for establishing your sites themes isn't it?

Brett_Tabke

3:00 pm on Mar 7, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Yes, but you can't neccessarily go looking for that page they spidered in the database. The inktomi/hotbot combo is currently king of throwing out pages while Excite ranks a close second.

priya

11:34 am on Mar 8, 2000 (gmt 0)

10+ Year Member



thanks for the info
by what brett says can i infer that spiders actively index all page contents while crawlers work more for link popularity algo.
what are fresh spiders and add url spider

priya

 

Featured Threads

Hot Threads This Week

Hot Threads This Month