Fast badly needs more updates

[Bobby_Davro]

I have been watching the listings for some of our sites in Fast/Alltheweb for some some time, and IMHO, Fast is pretty poor at updating.

As an example, we closed down a site over 6 months ago, possibly even up to 9 months ago. Yet when I checked, Alltheweb still has 271 pages listed from the site.

Checking for one of our live sites which has been up for a year, with many thousands of pages, reveals just 70 pages. The same story applies to a number of our other sites. It almost looks like it is just the first layer of pages that have been indexed and the spider isn't following the links downward, despite a strong and clear linking structure on each site.

Surely it is time for Fast to up the pace of spidering?

[HenryUK]

Following some changes, my site was indexed properly (tens of thousands of pages) by FAST about three weeks ago, and it has since spidered and reindexed - which I can tell from (a) the number of pages on particular searches and (b) the freshness of some of the content.

However I agree that it does not seem to be too good at removing outdated pages - we have a churn rate of around 2500 pages a month (new data arriving and old data expiring)and it looks to me as though the old data is still there.

[creative craig]

I have a site that has only just been fully indexed by Fast in the last month or so, it has been online since last summer. The linking structure hasnt changed on the site since it was first went live.

It was fully indexed by Google, Altavista (which amazed me) and Inktomie all with in the first 3 months of the site going live... it just seemed that fast had a problem with the site.

Craig

[edited by: creative_craig at 10:01 am (utc) on Dec. 13, 2002]

[pendanticist]

Earlier today e-mail'd taskforce on this very issue. It would seem that the folks in Norway might just be preparing to do an update.

Pendanticist.

[Bobby_Davro]

At least one of the sites has been fully crawled and then not updated, which is even stranger. Perhaps we are doing something to upset the Fast spider, but it seems strange that they would index some pages and not others.