Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

fast crawler

where are you?



10:40 am on Nov 13, 2001 (gmt 0)

10+ Year Member

Has anyone managed to get new sites spidered by fast since the begining of October 2001?

All my existing clients' prior to Oct 2001 continue to be spidered on a bi-monthly-ish cycle, but new clients' have not.

Any suggestions?


3:09 pm on Nov 13, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

He will be there soon, just wait



5:22 pm on Nov 13, 2001 (gmt 0)

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Hello HIPO
Did you submit the sites, are there links pointing to them?


5:30 pm on Nov 13, 2001 (gmt 0)

10+ Year Member


I have both submitted the site to the add url page and they have links from existing sites in the fast database.


7:21 pm on Nov 14, 2001 (gmt 0)

10+ Year Member

HIPO, then it must come very fast. fastbot loves links. I had new sites indexed very recently, just by links, no submitting.

Doublecheck your robot.txt file...


12:32 am on Nov 15, 2001 (gmt 0)

I too am trying to find out why my sites don't get refreshed more often by the FAST webcrawler. This summer they announced that they are refreshing their index every 9-12 days, but even though my sites are well represented in their index, I don't get refreshed very often.

Here's what I've been seeing: a small (test?) crawl in September (9/7-12), a full crawl in late September (9/21-10/3), another crawl (with FAST-WebCrawler/3.3) in mid-October (10/19-26)), and what seems like a full crawl starting yesterday (11/13). So, basically one crawl every 4 weeks.

What is everyone else seeing? Do I have to do something special to be refreshed more often? Should I submit my URLs every 9-12 days?


10:01 am on Nov 15, 2001 (gmt 0)

10+ Year Member

starec, i was under the impression that fastbot loves links aswell.

all my sites prior to october get spidered on a regular basis, and a great number of these sites have links to my post october sites, but no joy.

we don't use a robot.txt file to exclude any spiders.


Featured Threads

Hot Threads This Week

Hot Threads This Month