Scaling up the Service

FAST is scaling up the web service, test crawler out

[KnutRisvik]

Dear Webmasters,

As you might have heard some rumours about, FAST is preparing a very significant service scaleup this fall. In order to do this, we are deploying a totally new and distributed crawler architecture. The new crawler has the ID of "FAST-WebCrawler/3.2 test", and is currently sweeping some final test runs. If anyone is troubled by this, please let me know and I will immediately forward to the testing team.

We hope our tests will bring you a significantly improved service this fall.

Best regards,

- Knut Magne

Knut Magne Risvik - [e-mail]
Director of Engineering
Fast Search & Transfer ASA

[IanTurner]

Many thanks for the information Knut.

Once again Fast proves that it is more concerned about its users than any of its competitors.

I have never been 'troubled' by the Fast robot (always a welcome sight in the logs) and I hope that your tests are successful.

[mark_roach]

Thanks for the info. Now if you could also tell us what its favourite food is, I will make sure I give it plenty to feed on ;)

[starec]

Nice to see some activity there, Knut. Keep the pressure up, I started thinking that it's becoming a one SE show...

[NFFC]

Maximum respect to the guy's from AllTheWeb :)

[bird]

The crawling intensity is sustained but still gentle. Looks like a really nice load balancing algo. It is rare to see a robot keep its request so consitently close to the recommended one-minute interval.

On the downside, I find that at least between Sept. 17th and 19th, the disallow lines in my robots.txt file were ignored by the test spider. No trespassing observed since Sept 20th though, so it may have been a temporary problem.

Good luck getting it to final!

[mayor]

Looked in my logs today and saw that your spider has been visiting, Knut. Well behaved and welcome. Keep up the good work.

[Brett_Tabke]

Thanks Knut.

I just looked at all our logs for the last couple of days. Requests are spread out nicely and most are in "off peak" hours.

[caine]

Thanks Knut,

I was getting worried, that the fast bot was'nt coming, but its happy come, and crawled.

Always the best place to inform first !

[pete]

Thanks for keeping us in the picture Knut.

[engine]

Excellent, thanks Knut.

I'm eagerly awaiting the test crawler, although, it seems to be FAST-WebCrawler/2.2.10 so far.

[littleman]

Thank you Knut for the info.

Here is what I have in my spider log:
HTTP_USER_AGENT = FAST-WebCrawler/3.2 test
REMOTE_ADDR = 66.77.74.214
Name: cr011r01-test.sac2.fastsearch.net

[volatilegx]

The FAST test crawler is hammering my site with bad URL requests... I wonder if they have a bug in the crawling routine. It seems they got a 404 error on a URL(oops) and then tried to append the directory name onto the end of the URL... to infinity.

The IP of the crawler is 66.77.74.208 and the UA is "FAST-WebCrawler/3.2 test"

[mr_dredd2]

I have to say that although FAST's contributions here are extremely encouraging, their spider has been, in my experience, not the most intelligent spider i've seen.

1) For many sites I manage, it just doesn't like removing pages from its index. I have loads of pages, no longer on the server, no links to them.. and they persist.

2) If you change the index page, i.e. change the name of the index page that the server serves up, it will not recognise it - it just keeps trying to find the old one and then disapearing.

I'm really appreciative of FAST sharing their plans with us. I'm also glad that their spider is being upgraded. I just hope that they put a bit more "intelligence" into it!!

[JuniorHarris]

The new test spider has done a great job...according to our logs it is executing page requests at almost exactly one minute apart!~ I really like Google, but I have to admit that FAST is much better at performing ANDed search criteria results (IMO)...however Google may still have a better spam filter. Bravo to FAST...would love to see them put some pressure on Google, so they are not the only game in town.

[lazerzubb]

I am wonder if you have done something special because i signed up for Partnersite program BETA have you started it yet and so what does it make dirrerence against before?