fast crawler

where are you?

[HIPO]

Has anyone managed to get new sites spidered by fast since the begining of October 2001?

All my existing clients' prior to Oct 2001 continue to be spidered on a bi-monthly-ish cycle, but new clients' have not.

Any suggestions?

[Ove]

He will be there soon, just wait

/Ove

[heini]

Hello HIPO
Did you submit the sites, are there links pointing to them?

[HIPO]

heini

I have both submitted the site to the add url page and they have links from existing sites in the fast database.

[starec]

HIPO, then it must come very fast. fastbot loves links. I had new sites indexed very recently, just by links, no submitting.

Doublecheck your robot.txt file...

[webmastahus]

I too am trying to find out why my sites don't get refreshed more often by the FAST webcrawler. This summer they announced that they are refreshing their index every 9-12 days, but even though my sites are well represented in their index, I don't get refreshed very often.

Here's what I've been seeing: a small (test?) crawl in September (9/7-12), a full crawl in late September (9/21-10/3), another crawl (with FAST-WebCrawler/3.3) in mid-October (10/19-26)), and what seems like a full crawl starting yesterday (11/13). So, basically one crawl every 4 weeks.

What is everyone else seeing? Do I have to do something special to be refreshed more often? Should I submit my URLs every 9-12 days?

[HIPO]

starec, i was under the impression that fastbot loves links aswell.

all my sites prior to october get spidered on a regular basis, and a great number of these sites have links to my post october sites, but no joy.

we don't use a robot.txt file to exclude any spiders.