homepage Welcome to WebmasterWorld Guest from 54.81.170.186
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Accredited PayPal World Seller

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
types of spiders
priya




msg:407615
 11:28 am on Mar 7, 2000 (gmt 0)

hi
under the spider spotting chart of searchenginewatch.com i came across three categories of spider
normal spider
instant spider
mega spider
how are they classified? what is the difference. does anyone know?

thanks in advance
priya

 

Brett_Tabke




msg:407616
 11:50 am on Mar 7, 2000 (gmt 0)

There are instant spiders, but they are currently a joke. (alta's and Lyco's add url spiders - but neither actually add the page). Mega spider is just a Crawler. There are a few left, but they act sporadically and don't index all content. All the current crop of major se's have full crawlers, but they don't index all content. Most often the full crawlers just raid links for link popularity algorythms and words to determine your sites theme - then they throw the page out of the database.

Determined




msg:407617
 1:32 pm on Mar 7, 2000 (gmt 0)

"Most often the full crawlers just raid links for link popularity algorythms and words to determine your sites theme - then they throw the page out of the database"

But it's still important to get spidered by these guys for establishing your sites themes isn't it?

Brett_Tabke




msg:407618
 3:00 pm on Mar 7, 2000 (gmt 0)

Yes, but you can't neccessarily go looking for that page they spidered in the database. The inktomi/hotbot combo is currently king of throwing out pages while Excite ranks a close second.

priya




msg:407619
 11:34 am on Mar 8, 2000 (gmt 0)

thanks for the info
by what brett says can i infer that spiders actively index all page contents while crawlers work more for link popularity algo.
what are fresh spiders and add url spider

priya

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved