homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

Scaling up the Service
FAST is scaling up the web service, test crawler out

 1:37 pm on Sep 24, 2001 (gmt 0)

Dear Webmasters,

As you might have heard some rumours about, FAST is preparing a very significant service scaleup this fall. In order to do this, we are deploying a totally new and distributed crawler architecture. The new crawler has the ID of "FAST-WebCrawler/3.2 test", and is currently sweeping some final test runs. If anyone is troubled by this, please let me know and I will immediately forward to the testing team.

We hope our tests will bring you a significantly improved service this fall.

Best regards,

- Knut Magne

Knut Magne Risvik - kmr@fast.no
Director of Engineering
Fast Search & Transfer ASA



 1:50 pm on Sep 24, 2001 (gmt 0)

Many thanks for the information Knut.

Once again Fast proves that it is more concerned about its users than any of its competitors.

I have never been 'troubled' by the Fast robot (always a welcome sight in the logs) and I hope that your tests are successful.


 3:00 pm on Sep 24, 2001 (gmt 0)

Thanks for the info. Now if you could also tell us what its favourite food is, I will make sure I give it plenty to feed on ;)


 8:13 am on Sep 25, 2001 (gmt 0)

Nice to see some activity there, Knut. Keep the pressure up, I started thinking that it's becoming a one SE show...


 8:55 am on Sep 25, 2001 (gmt 0)

Maximum respect to the guy's from AllTheWeb :)


 11:47 pm on Sep 25, 2001 (gmt 0)

The crawling intensity is sustained but still gentle. Looks like a really nice load balancing algo. It is rare to see a robot keep its request so consitently close to the recommended one-minute interval.

On the downside, I find that at least between Sept. 17th and 19th, the disallow lines in my robots.txt file were ignored by the test spider. No trespassing observed since Sept 20th though, so it may have been a temporary problem.

Good luck getting it to final!


 2:45 am on Sep 27, 2001 (gmt 0)

Looked in my logs today and saw that your spider has been visiting, Knut. Well behaved and welcome. Keep up the good work.


 1:31 pm on Sep 27, 2001 (gmt 0)

Thanks Knut.

I just looked at all our logs for the last couple of days. Requests are spread out nicely and most are in "off peak" hours.


 2:10 pm on Sep 27, 2001 (gmt 0)

Thanks Knut,

I was getting worried, that the fast bot was'nt coming, but its happy come, and crawled.

Always the best place to inform first !


 3:23 pm on Sep 27, 2001 (gmt 0)

Thanks for keeping us in the picture Knut.


 3:31 pm on Sep 27, 2001 (gmt 0)

Excellent, thanks Knut.

I'm eagerly awaiting the test crawler, although, it seems to be FAST-WebCrawler/2.2.10 so far.


 6:54 pm on Sep 27, 2001 (gmt 0)

Thank you Knut for the info.

Here is what I have in my spider log:
HTTP_USER_AGENT = FAST-WebCrawler/3.2 test
Name: cr011r01-test.sac2.fastsearch.net


 7:10 pm on Sep 27, 2001 (gmt 0)

The FAST test crawler is hammering my site with bad URL requests... I wonder if they have a bug in the crawling routine. It seems they got a 404 error on a URL(oops) and then tried to append the directory name onto the end of the URL... to infinity.

The IP of the crawler is and the UA is "FAST-WebCrawler/3.2 test"


 8:26 am on Sep 28, 2001 (gmt 0)

I have to say that although FAST's contributions here are extremely encouraging, their spider has been, in my experience, not the most intelligent spider i've seen.

1) For many sites I manage, it just doesn't like removing pages from its index. I have loads of pages, no longer on the server, no links to them.. and they persist.

2) If you change the index page, i.e. change the name of the index page that the server serves up, it will not recognise it - it just keeps trying to find the old one and then disapearing.

I'm really appreciative of FAST sharing their plans with us. I'm also glad that their spider is being upgraded. I just hope that they put a bit more "intelligence" into it!!


 4:55 pm on Sep 28, 2001 (gmt 0)

The new test spider has done a great job...according to our logs it is executing page requests at almost exactly one minute apart!~ I really like Google, but I have to admit that FAST is much better at performing ANDed search criteria results (IMO)...however Google may still have a better spam filter. Bravo to FAST...would love to see them put some pressure on Google, so they are not the only game in town.


 5:16 pm on Sep 28, 2001 (gmt 0)

I am wonder if you have done something special because i signed up for Partnersite program BETA have you started it yet and so what does it make dirrerence against before?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved