Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: open
As an example, we closed down a site over 6 months ago, possibly even up to 9 months ago. Yet when I checked, Alltheweb still has 271 pages listed from the site.
Checking for one of our live sites which has been up for a year, with many thousands of pages, reveals just 70 pages. The same story applies to a number of our other sites. It almost looks like it is just the first layer of pages that have been indexed and the spider isn't following the links downward, despite a strong and clear linking structure on each site.
Surely it is time for Fast to up the pace of spidering?
However I agree that it does not seem to be too good at removing outdated pages - we have a churn rate of around 2500 pages a month (new data arriving and old data expiring)and it looks to me as though the old data is still there.
It was fully indexed by Google, Altavista (which amazed me) and Inktomie all with in the first 3 months of the site going live... it just seemed that fast had a problem with the site.
[edited by: creative_craig at 10:01 am (utc) on Dec. 13, 2002]