FAST / ATW update cycles

every 7 - 11 days?

[heini]

It was hard over the last months to see a pattern in FAST's update cycles. In fact it was hard to see cycles...
Now that's not surprising, given the massive 150% growth of the index over the last quarter, from some 800 Mill to more than 2 Bill pages.

Before this massive expansion, FAST aimed at refresh cycles of 9 - 12 days.

A new press release [fast.no] claims a complete update every 7 - 11 days. If FAST could meet that, it surely would put them ahead significantly of the competition.

Will be interesting to see if this holds true.

[Rumbas]

So we're talking up to 3 full refreshes a month?
That would certainly make them a contender for the Y! spot and stand out from the rest - if they can pull it off.

Very interesting for sure.

[vitaplease]

Interesting,

just imagine the amount of spidering they will have to do to keep that up-to-date.

Great, nothing better than more good competition around.

[caine]

with an index revision on that regularity, they can play seriously with there algo's, to get it right. Due to the bad idea's being backtrackable in such a short time for the next update, they can work on a hit and miss principle and end up with killer relevancies, that will surely shift the balance of the SE stakes, and ultimately their value to the big money players in the game.

Can't see how they will go wrong!

[Abrexa_UK]

I don't see it myself. Fast did a massive crawl of our directory at the same time as Google, and finished on the 7th (IIRC). It is now the 18th (11 days), and nothing has appeared.

It also features around 200 pages from the site that haven't existed for around 2-3 weeks.

This doesn't seem consistent with a 7-11 day refresh cycle.

[heini]

There's one interesting sentence in that Reg. article [webmasterworld.com]:

Fast says a large portion of its index, the popular content, is re- spidered every seven to 11 days, with the remainder being spidered every month or so

Apparently it was Jami Axelroed, ATW senior product manager, who said that.

Now, I'd be very interested in how FAST defines what is popular content...?

[edited by: heini at 7:11 pm (utc) on June 18, 2002]

[lazerzubb]

popular content...?

Last chat i had about this was that they spidered pages that had new content often, who got many clicks (remember they can track which sites get traffic via alltheweb's logfile) and pages with a lot of inbound links, i am not so sure about the last one though.

[EliteWeb]

Alright guys n' girls ;) I wanna help out all the engines so I put up a site that is being crawled by them. I dont care which one spiders my site first so their webpage shows they have a higher page count, oOoo no i dont ;)

Congrats to Fast, too bad I dont use them all that much.

[heini]

> who got many clicks (remember they can track which sites get traffic via alltheweb's logfile)

Can't really imagine that's true. ATW is a specialists engine. Measuring clickpop from ATW would very likely produce a rather distorted picture of what the mass market wants. Add the fact that ATW is only available with an English interface...

>had new content often,
possible. FAST's spider armada often spiders pages en masse without neccesarily refreshing them in the index.

>and pages with a lot of inbound links
Hmm. Would make more sense if refreshing would be triggered by discovering new links.

Knut Magne Risvik, Director of Engineering at FAST, has said in this thread [webmasterworld.com], FAST aims at

being able to cover the most important parts of the web.... that's where our focus lies.

So what's most important then?
FAST answers millions of queries daily, through their partners like the worldwide Lycos network. They sure have lists of most popular search terms from all over the world and many different languages.

What kind of sites do get refreshed every 7-11 days? Are they in a different db from the sites, that get rather montly updates?
What percentage of the 2,1 Bill. pages is covered by the short update cycle?

[lazerzubb]

Sure ATW is not a big engine but it will give you some overview over which sites get more clicks than others, and i am sure they can access Lycos log file etc, but i don't know if they use this it's just something i had back in my head.