Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

FAST crawler gets faster

Thats what i like to hear.....

5:53 pm on Jul 23, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:July 27, 2000
votes: 0


"Fast Search & Transfer has upgraded the crawlers of its AlltheWeb.com search engine to revisit all 625m pages of its index every 9 to 12 days"

6:22 pm on July 23, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 16

Oh dear. I like up to date SE databases. However, Altavista and Fast are starting to put a real strain on things. The number of requests they can make in a week is getting into the, do I ban them territory. (*sigh* I can't believe I just said that).

I really feel for a former client - 75k pages on two domains (tech manuals). Between the major search engines, he is averaging 200k views a week from the engines. Throw in rogue spiders and it jumps to 500k a week (100gig a month in bandwidth just for spiders).

The litmus test:
If a search engine does more spider page views than it sends out referrals.

6:35 pm on July 23, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 7, 2000
votes: 0

>The litmus test:
>If a search engine does more spider page views than it sends out referrals.

Fast will deliver results "round trip" in 600 ms
The question then becomes "how long does the spider take?"

The question then becomes how relivant is your site to the query AND "how long does the spider take?"


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members