homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

FAST / ATW update cycles
every 7 - 11 days?

 9:41 am on Jun 17, 2002 (gmt 0)

It was hard over the last months to see a pattern in FAST's update cycles. In fact it was hard to see cycles...
Now that's not surprising, given the massive 150% growth of the index over the last quarter, from some 800 Mill to more than 2 Bill pages.

Before this massive expansion, FAST aimed at refresh cycles of 9 - 12 days.

A new press release [fast.no] claims a complete update every 7 - 11 days. If FAST could meet that, it surely would put them ahead significantly of the competition.

Will be interesting to see if this holds true.



 11:12 am on Jun 17, 2002 (gmt 0)

So we're talking up to 3 full refreshes a month?
That would certainly make them a contender for the Y! spot and stand out from the rest - if they can pull it off.

Very interesting for sure.


 11:19 am on Jun 17, 2002 (gmt 0)


just imagine the amount of spidering they will have to do to keep that up-to-date.

Great, nothing better than more good competition around.


 11:20 am on Jun 17, 2002 (gmt 0)

with an index revision on that regularity, they can play seriously with there algo's, to get it right. Due to the bad idea's being backtrackable in such a short time for the next update, they can work on a hit and miss principle and end up with killer relevancies, that will surely shift the balance of the SE stakes, and ultimately their value to the big money players in the game.

Can't see how they will go wrong!


 1:04 pm on Jun 18, 2002 (gmt 0)

I don't see it myself. Fast did a massive crawl of our directory at the same time as Google, and finished on the 7th (IIRC). It is now the 18th (11 days), and nothing has appeared.

It also features around 200 pages from the site that haven't existed for around 2-3 weeks.

This doesn't seem consistent with a 7-11 day refresh cycle.


 2:18 pm on Jun 18, 2002 (gmt 0)

There's one interesting sentence in that Reg. article [webmasterworld.com]:

Fast says a large portion of its index, the popular content, is re- spidered every seven to 11 days, with the remainder being spidered every month or so

Apparently it was Jami Axelroed, ATW senior product manager, who said that.

Now, I'd be very interested in how FAST defines what is popular content...?

[edited by: heini at 7:11 pm (utc) on June 18, 2002]


 2:17 pm on Jun 18, 2002 (gmt 0)

popular content...?

Last chat i had about this was that they spidered pages that had new content often, who got many clicks (remember they can track which sites get traffic via alltheweb's logfile) and pages with a lot of inbound links, i am not so sure about the last one though.


 4:22 pm on Jun 18, 2002 (gmt 0)

Alright guys n' girls ;) I wanna help out all the engines so I put up a site that is being crawled by them. I dont care which one spiders my site first so their webpage shows they have a higher page count, oOoo no i dont ;)

Congrats to Fast, too bad I dont use them all that much.


 8:24 pm on Jun 18, 2002 (gmt 0)

> who got many clicks (remember they can track which sites get traffic via alltheweb's logfile)

Can't really imagine that's true. ATW is a specialists engine. Measuring clickpop from ATW would very likely produce a rather distorted picture of what the mass market wants. Add the fact that ATW is only available with an English interface...

>had new content often,
possible. FAST's spider armada often spiders pages en masse without neccesarily refreshing them in the index.

>and pages with a lot of inbound links
Hmm. Would make more sense if refreshing would be triggered by discovering new links.

Knut Magne Risvik, Director of Engineering at FAST, has said in this thread [webmasterworld.com], FAST aims at
being able to cover the most important parts of the web.... that's where our focus lies.

So what's most important then?
FAST answers millions of queries daily, through their partners like the worldwide Lycos network. They sure have lists of most popular search terms from all over the world and many different languages.

What kind of sites do get refreshed every 7-11 days? Are they in a different db from the sites, that get rather montly updates?
What percentage of the 2,1 Bill. pages is covered by the short update cycle?


 6:09 am on Jun 19, 2002 (gmt 0)

Sure ATW is not a big engine but it will give you some overview over which sites get more clicks than others, and i am sure they can access Lycos log file etc, but i don't know if they use this it's just something i had back in my head.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved