homepage Welcome to WebmasterWorld Guest from 54.242.126.9
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
fast crawler
where are you?
HIPO

10+ Year Member



 
Msg#: 354 posted 10:40 am on Nov 13, 2001 (gmt 0)

Has anyone managed to get new sites spidered by fast since the begining of October 2001?

All my existing clients' prior to Oct 2001 continue to be spidered on a bi-monthly-ish cycle, but new clients' have not.

Any suggestions?

 

Ove

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 354 posted 3:09 pm on Nov 13, 2001 (gmt 0)

He will be there soon, just wait

/Ove

heini

WebmasterWorld Senior Member heini us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 354 posted 5:22 pm on Nov 13, 2001 (gmt 0)

Hello HIPO
Did you submit the sites, are there links pointing to them?

HIPO

10+ Year Member



 
Msg#: 354 posted 5:30 pm on Nov 13, 2001 (gmt 0)

heini

I have both submitted the site to the add url page and they have links from existing sites in the fast database.

starec

10+ Year Member



 
Msg#: 354 posted 7:21 pm on Nov 14, 2001 (gmt 0)

HIPO, then it must come very fast. fastbot loves links. I had new sites indexed very recently, just by links, no submitting.

Doublecheck your robot.txt file...

webmastahus

10+ Year Member



 
Msg#: 354 posted 12:32 am on Nov 15, 2001 (gmt 0)

I too am trying to find out why my sites don't get refreshed more often by the FAST webcrawler. This summer they announced that they are refreshing their index every 9-12 days, but even though my sites are well represented in their index, I don't get refreshed very often.

Here's what I've been seeing: a small (test?) crawl in September (9/7-12), a full crawl in late September (9/21-10/3), another crawl (with FAST-WebCrawler/3.3) in mid-October (10/19-26)), and what seems like a full crawl starting yesterday (11/13). So, basically one crawl every 4 weeks.

What is everyone else seeing? Do I have to do something special to be refreshed more often? Should I submit my URLs every 9-12 days?

HIPO

10+ Year Member



 
Msg#: 354 posted 10:01 am on Nov 15, 2001 (gmt 0)

starec, i was under the impression that fastbot loves links aswell.

all my sites prior to october get spidered on a regular basis, and a great number of these sites have links to my post october sites, but no joy.

we don't use a robot.txt file to exclude any spiders.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved