Welcome to WebmasterWorld Guest from 107.20.75.63

Forum Moderators: open

Message Too Old, No Replies

frequency of fast spider

     
4:24 pm on Jul 5, 2001 (gmt 0)

Full Member

10+ Year Member

joined:June 14, 2001
posts:221
votes: 0


any ideas? it seemed to be about once a week a little while ago but has slacked off recently.

Air

5:46 pm on July 5, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 10, 2000
posts:1253
votes: 0


From my records, it was pretty much once a month, although the dates are not very regular, in June in showed up on the 6th, 9th, 10th, and 21st. of the month. Might want to read some on the threads on Pay for inclusion in this forum, I expect the spidering pattern to change.
10:16 am on July 10, 2001 (gmt 0)

Full Member

10+ Year Member

joined:Feb 19, 2001
posts:308
votes: 0


Looks like another deep crawl in progress right now.
9:42 am on July 16, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 15, 2000
posts:482
votes: 0


Spidercycles (# of spidered pages)
2nd - 9th of may (8890)/ update 16th - 17th of may
18th - 21st of may (3851)/ update 25th - 26th of may
25th - 31st of may (16745)/ update 4th - 5th of june
5th - 13th of june (29431)/ update 20th - 21st of june
21st - 28th of june (15222)/ update 1st - 2nd of july
now: 9th - 11th of july (3091) / update ??

So, as my question was in another thread - did anybody see the same thing that FAST doesnīt go through the hole cycle right now but cut it off?

[edit]
Looks like as I forgot an update:
with the relaunch last week around the 10th of july the numbers of pages in the index changed a bit. Didnīt notice a spidering between the end of june and the relaunch-update around the 10th.

8:43 pm on July 30, 2001 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
posts:2531
votes: 1


Just picked up these 2 from Webcrawler:

209.202.148.12 - - [30/Jul/2001:12:35:47 -0400] "GET /robots.txt HTTP/1.0" 200 134 "-" "FAST-WebCrawler/2.2.6 (crawler@fast.no; [fast.no...]

209.202.148.12 - - [30/Jul/2001:14:00:28 -0400] "GET /DIRECTORY/file.html HTTP/1.0" 304 - "-" "FAST-WebCrawler/2.2.6 (crawler@fast.no; [fast.no...]

BTW anyone know why it apparently starts at a random file?

4:40 am on July 31, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 15, 2000
posts:482
votes: 0


I often saw that happen. Our pages has been in the index before or there is a link from anywhere to that page. So, it has not necessarily to start at the root.
8:36 am on Aug 1, 2001 (gmt 0)

Full Member

10+ Year Member

joined:June 14, 2001
posts:221
votes: 0


2 of my sites I manage were previously spidered by fast and now the spider just will not go past trying to find the old homepage which is no longer there..
12:05 pm on Aug 1, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 21, 2001
posts:2488
votes: 0


Mr_Dredd2.

dropping links, such as the old home page is not a good idea.

I been waiting for Fast to crawl all of last month, then it finally came.

From my logs it seems the frequency of spidering is around a month.

12:24 pm on Aug 1, 2001 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
posts:2531
votes: 1


>Our pages has been in the index before

My pages have been in there before too, so that should explain it :)

3:03 pm on Aug 1, 2001 (gmt 0)

Full Member

10+ Year Member

joined:June 14, 2001
posts:221
votes: 0


my problem is that the old pages were a "legacy" from before i was involved. Any ideas how I can "kickstart" the fast spider to pick up the new index page?
(just subbing the url doesnt work)..

:)

2:36 pm on Aug 3, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2000
posts:920
votes: 0


I think this is preparation for the paid submission. They'll make it harder to get into the search engine, but have a similar system on the paid inclusion as it had. Fast is evolving too fast. It is going through the 4 years of Infoseek all in one year. Worrying...
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members