homepage Welcome to WebmasterWorld Guest from 54.161.192.130
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
FAST crawler gets faster
Thats what i like to hear.....
roscoepico

10+ Year Member



 
Msg#: 598 posted 5:53 pm on Jul 23, 2001 (gmt 0)

[netimperative.com...]

"Fast Search & Transfer has upgraded the crawlers of its AlltheWeb.com search engine to revisit all 625m pages of its index every 9 to 12 days"

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 598 posted 6:22 pm on Jul 23, 2001 (gmt 0)

Oh dear. I like up to date SE databases. However, Altavista and Fast are starting to put a real strain on things. The number of requests they can make in a week is getting into the, do I ban them territory. (*sigh* I can't believe I just said that).

I really feel for a former client - 75k pages on two domains (tech manuals). Between the major search engines, he is averaging 200k views a week from the engines. Throw in rogue spiders and it jumps to 500k a week (100gig a month in bandwidth just for spiders).

The litmus test:
If a search engine does more spider page views than it sends out referrals.

Mike_Mackin

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 598 posted 6:35 pm on Jul 23, 2001 (gmt 0)

>The litmus test:
>If a search engine does more spider page views than it sends out referrals.

Fast will deliver results "round trip" in 600 ms
The question then becomes "how long does the spider take?"

ADDED
The question then becomes how relivant is your site to the query AND "how long does the spider take?"

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved