Welcome to WebmasterWorld Guest from 54.161.99.20

Forum Moderators: open

Message Too Old, No Replies

Scaling up the Service

FAST is scaling up the web service, test crawler out

     
1:37 pm on Sep 24, 2001 (gmt 0)

New User

10+ Year Member

joined:Aug 22, 2001
posts:9
votes: 0


Dear Webmasters,

As you might have heard some rumours about, FAST is preparing a very significant service scaleup this fall. In order to do this, we are deploying a totally new and distributed crawler architecture. The new crawler has the ID of "FAST-WebCrawler/3.2 test", and is currently sweeping some final test runs. If anyone is troubled by this, please let me know and I will immediately forward to the testing team.

We hope our tests will bring you a significantly improved service this fall.

Best regards,

- Knut Magne

Knut Magne Risvik - kmr@fast.no
Director of Engineering
Fast Search & Transfer ASA

1:50 pm on Sept 24, 2001 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator ianturner is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 19, 2001
posts:3505
votes: 27


Many thanks for the information Knut.

Once again Fast proves that it is more concerned about its users than any of its competitors.

I have never been 'troubled' by the Fast robot (always a welcome sight in the logs) and I hope that your tests are successful.

3:00 pm on Sept 24, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:July 28, 2000
posts:580
votes: 0


Thanks for the info. Now if you could also tell us what its favourite food is, I will make sure I give it plenty to feed on ;)
8:13 am on Sept 25, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 17, 2001
posts:409
votes: 0


Nice to see some activity there, Knut. Keep the pressure up, I started thinking that it's becoming a one SE show...
8:55 am on Sept 25, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member nffc is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 22, 2000
posts:3604
votes: 0


Maximum respect to the guy's from AllTheWeb :)
11:47 pm on Sept 25, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 10, 2001
posts:1551
votes: 10


The crawling intensity is sustained but still gentle. Looks like a really nice load balancing algo. It is rare to see a robot keep its request so consitently close to the recommended one-minute interval.

On the downside, I find that at least between Sept. 17th and 19th, the disallow lines in my robots.txt file were ignored by the test spider. No trespassing observed since Sept 20th though, so it may have been a temporary problem.

Good luck getting it to final!

2:45 am on Sept 27, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 29, 2000
posts:1133
votes: 0


Looked in my logs today and saw that your spider has been visiting, Knut. Well behaved and welcome. Keep up the good work.
1:31 pm on Sept 27, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38059
votes: 13


Thanks Knut.

I just looked at all our logs for the last couple of days. Requests are spread out nicely and most are in "off peak" hours.

2:10 pm on Sept 27, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 21, 2001
posts:2489
votes: 0


Thanks Knut,

I was getting worried, that the fast bot was'nt coming, but its happy come, and crawled.

Always the best place to inform first !

3:23 pm on Sept 27, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 21, 1999
posts:370
votes: 0


Thanks for keeping us in the picture Knut.
3:31 pm on Sept 27, 2001 (gmt 0)

Administrator from GB 

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 9, 2000
posts:23042
votes: 332


Excellent, thanks Knut.

I'm eagerly awaiting the test crawler, although, it seems to be FAST-WebCrawler/2.2.10 so far.

6:54 pm on Sept 27, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
posts:2924
votes: 0


Thank you Knut for the info.

Here is what I have in my spider log:
HTTP_USER_AGENT = FAST-WebCrawler/3.2 test
REMOTE_ADDR = 66.77.74.214
Name: cr011r01-test.sac2.fastsearch.net

7:10 pm on Sept 27, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
posts:2450
votes: 0


The FAST test crawler is hammering my site with bad URL requests... I wonder if they have a bug in the crawling routine. It seems they got a 404 error on a URL(oops) and then tried to append the directory name onto the end of the URL... to infinity.

The IP of the crawler is 66.77.74.208 and the UA is "FAST-WebCrawler/3.2 test"

8:26 am on Sept 28, 2001 (gmt 0)

Full Member

10+ Year Member

joined:June 14, 2001
posts:221
votes: 0


I have to say that although FAST's contributions here are extremely encouraging, their spider has been, in my experience, not the most intelligent spider i've seen.

1) For many sites I manage, it just doesn't like removing pages from its index. I have loads of pages, no longer on the server, no links to them.. and they persist.

2) If you change the index page, i.e. change the name of the index page that the server serves up, it will not recognise it - it just keeps trying to find the old one and then disapearing.

I'm really appreciative of FAST sharing their plans with us. I'm also glad that their spider is being upgraded. I just hope that they put a bit more "intelligence" into it!!

4:55 pm on Sept 28, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 14, 2001
posts:129
votes: 0


The new test spider has done a great job...according to our logs it is executing page requests at almost exactly one minute apart!~ I really like Google, but I have to admit that FAST is much better at performing ANDed search criteria results (IMO)...however Google may still have a better spam filter. Bravo to FAST...would love to see them put some pressure on Google, so they are not the only game in town.
5:16 pm on Sept 28, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2001
posts:2059
votes: 0


I am wonder if you have done something special because i signed up for Partnersite program BETA have you started it yet and so what does it make dirrerence against before?