homepage Welcome to WebmasterWorld Guest from 54.234.74.85
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
Scaling up the Service
FAST is scaling up the web service, test crawler out
KnutRisvik




msg:225072
 1:37 pm on Sep 24, 2001 (gmt 0)

Dear Webmasters,

As you might have heard some rumours about, FAST is preparing a very significant service scaleup this fall. In order to do this, we are deploying a totally new and distributed crawler architecture. The new crawler has the ID of "FAST-WebCrawler/3.2 test", and is currently sweeping some final test runs. If anyone is troubled by this, please let me know and I will immediately forward to the testing team.

We hope our tests will bring you a significantly improved service this fall.

Best regards,

- Knut Magne

Knut Magne Risvik - kmr@fast.no
Director of Engineering
Fast Search & Transfer ASA

 

IanTurner




msg:225073
 1:50 pm on Sep 24, 2001 (gmt 0)

Many thanks for the information Knut.

Once again Fast proves that it is more concerned about its users than any of its competitors.

I have never been 'troubled' by the Fast robot (always a welcome sight in the logs) and I hope that your tests are successful.

mark_roach




msg:225074
 3:00 pm on Sep 24, 2001 (gmt 0)

Thanks for the info. Now if you could also tell us what its favourite food is, I will make sure I give it plenty to feed on ;)

starec




msg:225075
 8:13 am on Sep 25, 2001 (gmt 0)

Nice to see some activity there, Knut. Keep the pressure up, I started thinking that it's becoming a one SE show...

NFFC




msg:225076
 8:55 am on Sep 25, 2001 (gmt 0)

Maximum respect to the guy's from AllTheWeb :)

bird




msg:225077
 11:47 pm on Sep 25, 2001 (gmt 0)

The crawling intensity is sustained but still gentle. Looks like a really nice load balancing algo. It is rare to see a robot keep its request so consitently close to the recommended one-minute interval.

On the downside, I find that at least between Sept. 17th and 19th, the disallow lines in my robots.txt file were ignored by the test spider. No trespassing observed since Sept 20th though, so it may have been a temporary problem.

Good luck getting it to final!

mayor




msg:225078
 2:45 am on Sep 27, 2001 (gmt 0)

Looked in my logs today and saw that your spider has been visiting, Knut. Well behaved and welcome. Keep up the good work.

Brett_Tabke




msg:225079
 1:31 pm on Sep 27, 2001 (gmt 0)

Thanks Knut.

I just looked at all our logs for the last couple of days. Requests are spread out nicely and most are in "off peak" hours.

caine




msg:225080
 2:10 pm on Sep 27, 2001 (gmt 0)

Thanks Knut,

I was getting worried, that the fast bot was'nt coming, but its happy come, and crawled.

Always the best place to inform first !

pete




msg:225081
 3:23 pm on Sep 27, 2001 (gmt 0)

Thanks for keeping us in the picture Knut.

engine




msg:225082
 3:31 pm on Sep 27, 2001 (gmt 0)

Excellent, thanks Knut.

I'm eagerly awaiting the test crawler, although, it seems to be FAST-WebCrawler/2.2.10 so far.

littleman




msg:225083
 6:54 pm on Sep 27, 2001 (gmt 0)

Thank you Knut for the info.

Here is what I have in my spider log:
HTTP_USER_AGENT = FAST-WebCrawler/3.2 test
REMOTE_ADDR = 66.77.74.214
Name: cr011r01-test.sac2.fastsearch.net

volatilegx




msg:225084
 7:10 pm on Sep 27, 2001 (gmt 0)

The FAST test crawler is hammering my site with bad URL requests... I wonder if they have a bug in the crawling routine. It seems they got a 404 error on a URL(oops) and then tried to append the directory name onto the end of the URL... to infinity.

The IP of the crawler is 66.77.74.208 and the UA is "FAST-WebCrawler/3.2 test"

mr_dredd2




msg:225085
 8:26 am on Sep 28, 2001 (gmt 0)

I have to say that although FAST's contributions here are extremely encouraging, their spider has been, in my experience, not the most intelligent spider i've seen.

1) For many sites I manage, it just doesn't like removing pages from its index. I have loads of pages, no longer on the server, no links to them.. and they persist.

2) If you change the index page, i.e. change the name of the index page that the server serves up, it will not recognise it - it just keeps trying to find the old one and then disapearing.

I'm really appreciative of FAST sharing their plans with us. I'm also glad that their spider is being upgraded. I just hope that they put a bit more "intelligence" into it!!

JuniorHarris




msg:225086
 4:55 pm on Sep 28, 2001 (gmt 0)

The new test spider has done a great job...according to our logs it is executing page requests at almost exactly one minute apart!~ I really like Google, but I have to admit that FAST is much better at performing ANDed search criteria results (IMO)...however Google may still have a better spam filter. Bravo to FAST...would love to see them put some pressure on Google, so they are not the only game in town.

lazerzubb




msg:225087
 5:16 pm on Sep 28, 2001 (gmt 0)

I am wonder if you have done something special because i signed up for Partnersite program BETA have you started it yet and so what does it make dirrerence against before?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved