frequency of fast spider

[mr_dredd2]

any ideas? it seemed to be about once a week a little while ago but has slacked off recently.

[Air]

From my records, it was pretty much once a month, although the dates are not very regular, in June in showed up on the 6th, 9th, 10th, and 21st. of the month. Might want to read some on the threads on Pay for inclusion in this forum, I expect the spidering pattern to change.

[FreeBee]

Looks like another deep crawl in progress right now.

[oLeon]

Spidercycles (# of spidered pages)
2nd - 9th of may (8890)/ update 16th - 17th of may
18th - 21st of may (3851)/ update 25th - 26th of may
25th - 31st of may (16745)/ update 4th - 5th of june
5th - 13th of june (29431)/ update 20th - 21st of june
21st - 28th of june (15222)/ update 1st - 2nd of july
now: 9th - 11th of july (3091) / update ??

So, as my question was in another thread - did anybody see the same thing that FAST doesnīt go through the hole cycle right now but cut it off?

[edit]
Looks like as I forgot an update:
with the relaunch last week around the 10th of july the numbers of pages in the index changed a bit. Didnīt notice a spidering between the end of june and the relaunch-update around the 10th.

[Rumbas]

Just picked up these 2 from Webcrawler:

209.202.148.12 - - [30/Jul/2001:12:35:47 -0400] "GET /robots.txt HTTP/1.0" 200 134 "-" "FAST-WebCrawler/2.2.6 ([e-mail]; [fast.no...]

209.202.148.12 - - [30/Jul/2001:14:00:28 -0400] "GET /DIRECTORY/file.html HTTP/1.0" 304 - "-" "FAST-WebCrawler/2.2.6 ([e-mail]; [fast.no...]

BTW anyone know why it apparently starts at a random file?

[oLeon]

I often saw that happen. Our pages has been in the index before or there is a link from anywhere to that page. So, it has not necessarily to start at the root.

[mr_dredd2]

2 of my sites I manage were previously spidered by fast and now the spider just will not go past trying to find the old homepage which is no longer there..

[caine]

Mr_Dredd2.

dropping links, such as the old home page is not a good idea.

I been waiting for Fast to crawl all of last month, then it finally came.

From my logs it seems the frequency of spidering is around a month.

[Rumbas]

>Our pages has been in the index before

My pages have been in there before too, so that should explain it :)

[mr_dredd2]

my problem is that the old pages were a "legacy" from before i was involved. Any ideas how I can "kickstart" the fast spider to pick up the new index page?
(just subbing the url doesnt work)..

:)

[backus]

I think this is preparation for the paid submission. They'll make it harder to get into the search engine, but have a similar system on the paid inclusion as it had. Fast is evolving too fast. It is going through the 4 years of Infoseek all in one year. Worrying...