Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

fast crawler

where are you?

10:40 am on Nov 13, 2001 (gmt 0)

New User

10+ Year Member

joined:Oct 12, 2000
votes: 0

Has anyone managed to get new sites spidered by fast since the begining of October 2001?

All my existing clients' prior to Oct 2001 continue to be spidered on a bi-monthly-ish cycle, but new clients' have not.

Any suggestions?


3:09 pm on Nov 13, 2001 (gmt 0)

Senior Member from SE 

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 24, 2001
votes: 0

He will be there soon, just wait


5:22 pm on Nov 13, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
votes: 0

Hello HIPO
Did you submit the sites, are there links pointing to them?
5:30 pm on Nov 13, 2001 (gmt 0)

New User

10+ Year Member

joined:Oct 12, 2000
votes: 0


I have both submitted the site to the add url page and they have links from existing sites in the fast database.

7:21 pm on Nov 14, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 17, 2001
votes: 0

HIPO, then it must come very fast. fastbot loves links. I had new sites indexed very recently, just by links, no submitting.

Doublecheck your robot.txt file...


12:32 am on Nov 15, 2001 (gmt 0)

Inactive Member
Account Expired


I too am trying to find out why my sites don't get refreshed more often by the FAST webcrawler. This summer they announced that they are refreshing their index every 9-12 days, but even though my sites are well represented in their index, I don't get refreshed very often.

Here's what I've been seeing: a small (test?) crawl in September (9/7-12), a full crawl in late September (9/21-10/3), another crawl (with FAST-WebCrawler/3.3) in mid-October (10/19-26)), and what seems like a full crawl starting yesterday (11/13). So, basically one crawl every 4 weeks.

What is everyone else seeing? Do I have to do something special to be refreshed more often? Should I submit my URLs every 9-12 days?

10:01 am on Nov 15, 2001 (gmt 0)

New User

10+ Year Member

joined:Oct 12, 2000
votes: 0

starec, i was under the impression that fastbot loves links aswell.

all my sites prior to october get spidered on a regular basis, and a great number of these sites have links to my post october sites, but no joy.

we don't use a robot.txt file to exclude any spiders.


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members