homepage Welcome to WebmasterWorld Guest from 54.237.98.229
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
Fast Spider
How often does fast spider sites?
VinceI

10+ Year Member



 
Msg#: 969 posted 4:36 pm on Nov 11, 2003 (gmt 0)

I just started the FAST WebCrawler spider on my site this month, but they are spidering my site every day with about 6000 hits a day.

I can't imagine that there is anything left to crawl...

How often does FAST usually spider a site?

How soon after they hit a site do you start showing up in their search engine results (just curious)?

 

heini

WebmasterWorld Senior Member heini us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 969 posted 10:33 pm on Nov 14, 2003 (gmt 0)

>I just started the FAST WebCrawler spider
How? Did you pay? Free Submit? And which spider do you get, FastCrawler, or FastPartnersite, or FirstPage?

VinceI

10+ Year Member



 
Msg#: 969 posted 4:57 pm on Dec 2, 2003 (gmt 0)

Most are from:

FAST-WebCrawler/3.8+(atw-crawler+at+fast+dot+no;+http://fast.no/support/crawler.asp)

Although it has been steadily declining, Urchin, my log analysis software, still reports about 3-4000 hits a day from FAST.

I didn't submit to them at all recently. I think I did the free submit thing some time in the past, but I definitely didn't pay for my submission.

So, is this unusual?

Rumbas

WebmasterWorld Administrator 10+ Year Member



 
Msg#: 969 posted 6:00 pm on Dec 2, 2003 (gmt 0)

>3-4000 hits

To me that's a bit unusual, does the site have a_lot of pages?

VinceI

10+ Year Member



 
Msg#: 969 posted 6:56 pm on Dec 2, 2003 (gmt 0)

About 3000 pages, plus my forums, but I have my robots.txt set up so that my forum isn't indexed (my site seemed to slow down too much when google crawled my forums).

I emailed them a few weeks ago about it, but didn't hear back.

I guess I'll try again if it continues this month.

Do they usually just crawl once a month or so?

steveb

WebmasterWorld Senior Member steveb us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 969 posted 2:35 am on Dec 3, 2003 (gmt 0)

You can be spidered daily, but in this case there is no guarantee those pages being spidered will ever really be included in the database. That Fast spider hits me all the time, but zero new pages have been added to the index in months and months. Hotbot shows a dozen of my pages last spidered in "December 31, 1969". Basically these look like all the pages I added this year. FAST crawls new pages all the time, but never adds them.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved