homepage Welcome to WebmasterWorld Guest from 54.204.215.209
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld
Visit PubCon.com
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
frequency of fast spider
mr_dredd2




msg:221521
 4:24 pm on Jul 5, 2001 (gmt 0)

any ideas? it seemed to be about once a week a little while ago but has slacked off recently.

 

Air




msg:221522
 5:46 pm on Jul 5, 2001 (gmt 0)

From my records, it was pretty much once a month, although the dates are not very regular, in June in showed up on the 6th, 9th, 10th, and 21st. of the month. Might want to read some on the threads on Pay for inclusion in this forum, I expect the spidering pattern to change.

FreeBee




msg:221523
 10:16 am on Jul 10, 2001 (gmt 0)

Looks like another deep crawl in progress right now.

oLeon




msg:221524
 9:42 am on Jul 16, 2001 (gmt 0)

Spidercycles (# of spidered pages)
2nd - 9th of may (8890)/ update 16th - 17th of may
18th - 21st of may (3851)/ update 25th - 26th of may
25th - 31st of may (16745)/ update 4th - 5th of june
5th - 13th of june (29431)/ update 20th - 21st of june
21st - 28th of june (15222)/ update 1st - 2nd of july
now: 9th - 11th of july (3091) / update ??

So, as my question was in another thread - did anybody see the same thing that FAST doesnīt go through the hole cycle right now but cut it off?

[edit]
Looks like as I forgot an update:
with the relaunch last week around the 10th of july the numbers of pages in the index changed a bit. Didnīt notice a spidering between the end of june and the relaunch-update around the 10th.

Rumbas




msg:221525
 8:43 pm on Jul 30, 2001 (gmt 0)

Just picked up these 2 from Webcrawler:

209.202.148.12 - - [30/Jul/2001:12:35:47 -0400] "GET /robots.txt HTTP/1.0" 200 134 "-" "FAST-WebCrawler/2.2.6 (crawler@fast.no; [fast.no...]

209.202.148.12 - - [30/Jul/2001:14:00:28 -0400] "GET /DIRECTORY/file.html HTTP/1.0" 304 - "-" "FAST-WebCrawler/2.2.6 (crawler@fast.no; [fast.no...]

BTW anyone know why it apparently starts at a random file?

oLeon




msg:221526
 4:40 am on Jul 31, 2001 (gmt 0)

I often saw that happen. Our pages has been in the index before or there is a link from anywhere to that page. So, it has not necessarily to start at the root.

mr_dredd2




msg:221527
 8:36 am on Aug 1, 2001 (gmt 0)

2 of my sites I manage were previously spidered by fast and now the spider just will not go past trying to find the old homepage which is no longer there..

caine




msg:221528
 12:05 pm on Aug 1, 2001 (gmt 0)

Mr_Dredd2.

dropping links, such as the old home page is not a good idea.

I been waiting for Fast to crawl all of last month, then it finally came.

From my logs it seems the frequency of spidering is around a month.

Rumbas




msg:221529
 12:24 pm on Aug 1, 2001 (gmt 0)

>Our pages has been in the index before

My pages have been in there before too, so that should explain it :)

mr_dredd2




msg:221530
 3:03 pm on Aug 1, 2001 (gmt 0)

my problem is that the old pages were a "legacy" from before i was involved. Any ideas how I can "kickstart" the fast spider to pick up the new index page?
(just subbing the url doesnt work)..

:)

backus




msg:221531
 2:36 pm on Aug 3, 2001 (gmt 0)

I think this is preparation for the paid submission. They'll make it harder to get into the search engine, but have a similar system on the paid inclusion as it had. Fast is evolving too fast. It is going through the 4 years of Infoseek all in one year. Worrying...

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved