homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

fast crawler
where are you?

 10:40 am on Nov 13, 2001 (gmt 0)

Has anyone managed to get new sites spidered by fast since the begining of October 2001?

All my existing clients' prior to Oct 2001 continue to be spidered on a bi-monthly-ish cycle, but new clients' have not.

Any suggestions?



 3:09 pm on Nov 13, 2001 (gmt 0)

He will be there soon, just wait



 5:22 pm on Nov 13, 2001 (gmt 0)

Hello HIPO
Did you submit the sites, are there links pointing to them?


 5:30 pm on Nov 13, 2001 (gmt 0)


I have both submitted the site to the add url page and they have links from existing sites in the fast database.


 7:21 pm on Nov 14, 2001 (gmt 0)

HIPO, then it must come very fast. fastbot loves links. I had new sites indexed very recently, just by links, no submitting.

Doublecheck your robot.txt file...


 12:32 am on Nov 15, 2001 (gmt 0)

I too am trying to find out why my sites don't get refreshed more often by the FAST webcrawler. This summer they announced that they are refreshing their index every 9-12 days, but even though my sites are well represented in their index, I don't get refreshed very often.

Here's what I've been seeing: a small (test?) crawl in September (9/7-12), a full crawl in late September (9/21-10/3), another crawl (with FAST-WebCrawler/3.3) in mid-October (10/19-26)), and what seems like a full crawl starting yesterday (11/13). So, basically one crawl every 4 weeks.

What is everyone else seeing? Do I have to do something special to be refreshed more often? Should I submit my URLs every 9-12 days?


 10:01 am on Nov 15, 2001 (gmt 0)

starec, i was under the impression that fastbot loves links aswell.

all my sites prior to october get spidered on a regular basis, and a great number of these sites have links to my post october sites, but no joy.

we don't use a robot.txt file to exclude any spiders.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved