Welcome to WebmasterWorld Guest from 54.163.68.15

Forum Moderators: open

Message Too Old, No Replies

Scaling up the Service

FAST is scaling up the web service, test crawler out

   
1:37 pm on Sep 24, 2001 (gmt 0)

10+ Year Member



Dear Webmasters,

As you might have heard some rumours about, FAST is preparing a very significant service scaleup this fall. In order to do this, we are deploying a totally new and distributed crawler architecture. The new crawler has the ID of "FAST-WebCrawler/3.2 test", and is currently sweeping some final test runs. If anyone is troubled by this, please let me know and I will immediately forward to the testing team.

We hope our tests will bring you a significantly improved service this fall.

Best regards,

- Knut Magne

Knut Magne Risvik - kmr@fast.no
Director of Engineering
Fast Search & Transfer ASA

1:50 pm on Sep 24, 2001 (gmt 0)

WebmasterWorld Administrator ianturner is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Many thanks for the information Knut.

Once again Fast proves that it is more concerned about its users than any of its competitors.

I have never been 'troubled' by the Fast robot (always a welcome sight in the logs) and I hope that your tests are successful.

3:00 pm on Sep 24, 2001 (gmt 0)

10+ Year Member



Thanks for the info. Now if you could also tell us what its favourite food is, I will make sure I give it plenty to feed on ;)
8:13 am on Sep 25, 2001 (gmt 0)

10+ Year Member



Nice to see some activity there, Knut. Keep the pressure up, I started thinking that it's becoming a one SE show...
8:55 am on Sep 25, 2001 (gmt 0)

WebmasterWorld Senior Member nffc is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Maximum respect to the guy's from AllTheWeb :)
11:47 pm on Sep 25, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The crawling intensity is sustained but still gentle. Looks like a really nice load balancing algo. It is rare to see a robot keep its request so consitently close to the recommended one-minute interval.

On the downside, I find that at least between Sept. 17th and 19th, the disallow lines in my robots.txt file were ignored by the test spider. No trespassing observed since Sept 20th though, so it may have been a temporary problem.

Good luck getting it to final!

2:45 am on Sep 27, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Looked in my logs today and saw that your spider has been visiting, Knut. Well behaved and welcome. Keep up the good work.
1:31 pm on Sep 27, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Thanks Knut.

I just looked at all our logs for the last couple of days. Requests are spread out nicely and most are in "off peak" hours.

2:10 pm on Sep 27, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Thanks Knut,

I was getting worried, that the fast bot was'nt coming, but its happy come, and crawled.

Always the best place to inform first !

3:23 pm on Sep 27, 2001 (gmt 0)

10+ Year Member



Thanks for keeping us in the picture Knut.
3:31 pm on Sep 27, 2001 (gmt 0)

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Excellent, thanks Knut.

I'm eagerly awaiting the test crawler, although, it seems to be FAST-WebCrawler/2.2.10 so far.

6:54 pm on Sep 27, 2001 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Thank you Knut for the info.

Here is what I have in my spider log:
HTTP_USER_AGENT = FAST-WebCrawler/3.2 test
REMOTE_ADDR = 66.77.74.214
Name: cr011r01-test.sac2.fastsearch.net

7:10 pm on Sep 27, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The FAST test crawler is hammering my site with bad URL requests... I wonder if they have a bug in the crawling routine. It seems they got a 404 error on a URL(oops) and then tried to append the directory name onto the end of the URL... to infinity.

The IP of the crawler is 66.77.74.208 and the UA is "FAST-WebCrawler/3.2 test"

8:26 am on Sep 28, 2001 (gmt 0)

10+ Year Member



I have to say that although FAST's contributions here are extremely encouraging, their spider has been, in my experience, not the most intelligent spider i've seen.

1) For many sites I manage, it just doesn't like removing pages from its index. I have loads of pages, no longer on the server, no links to them.. and they persist.

2) If you change the index page, i.e. change the name of the index page that the server serves up, it will not recognise it - it just keeps trying to find the old one and then disapearing.

I'm really appreciative of FAST sharing their plans with us. I'm also glad that their spider is being upgraded. I just hope that they put a bit more "intelligence" into it!!

4:55 pm on Sep 28, 2001 (gmt 0)

10+ Year Member



The new test spider has done a great job...according to our logs it is executing page requests at almost exactly one minute apart!~ I really like Google, but I have to admit that FAST is much better at performing ANDed search criteria results (IMO)...however Google may still have a better spam filter. Bravo to FAST...would love to see them put some pressure on Google, so they are not the only game in town.
5:16 pm on Sep 28, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I am wonder if you have done something special because i signed up for Partnersite program BETA have you started it yet and so what does it make dirrerence against before?