homepage Welcome to WebmasterWorld Guest from 23.22.59.252
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
The FAST WebCrawler Army
in fully coordinated attack mode
bird

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 408 posted 4:15 am on Feb 3, 2002 (gmt 0)

66.77.73.72 - - [01/Feb/2002:16:41:05 -0500] "GET /robots.txt HTTP/1.0" 200 183 "-" "FAST-WebCrawler/3.3 (crawler@fast.no; http_://fast.no/support.php?c=faqs/crawler)"
66.77.73.83 - - [01/Feb/2002:16:45:36 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.85 - - [01/Feb/2002:16:48:27 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.84 - - [01/Feb/2002:16:54:46 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.87 - - [01/Feb/2002:17:06:18 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.81 - - [01/Feb/2002:17:06:52 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.86 - - [01/Feb/2002:17:11:03 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.78 - - [01/Feb/2002:17:12:32 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.80 - - [01/Feb/2002:17:12:43 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.82 - - [01/Feb/2002:17:16:05 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.77 - - [01/Feb/2002:17:20:46 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.73 - - [01/Feb/2002:17:21:06 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.75 - - [01/Feb/2002:17:45:32 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.74 - - [01/Feb/2002:18:00:15 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.79 - - [01/Feb/2002:18:07:35 -0500] "GET /robots.txt HTTP/1.0"...
66.77.73.76 - - [01/Feb/2002:18:34:41 -0500] "GET /robots.txt HTTP/1.0"...

And that was only the beginning.

After that, they all started crawling my site, on an average half a dozen pages per minute, probably well past midnight, where my current log file ends. Looks like each one of those 16 robots fetched every single HTML document from my site seperately.

I wonder if they will now rank my site cumulatively in exchange? ;)

 

royalelephant

10+ Year Member



 
Msg#: 408 posted 4:49 pm on Feb 3, 2002 (gmt 0)

Me2

Yesterday, 020202, 12 individual FAST servers (66.77.73.*) made 12 visits to one of my little sites and each of the 12 FAST servers went to nearly half of my entire site. Is this normal behaviour for FAST or should I put in a command in robots.txt?

Pg Req Date ISP
272: 272: 6.97%: 2/Feb/02 04:51: 66.77.73.73
272: 272: 6.97%: 2/Feb/02 10:14: 66.77.73.74
272: 272: 6.97%: 2/Feb/02 07:32: 66.77.73.75
38: 38: 0.74%: 2/Feb/02 12:01: 66.77.73.76
272: 272: 6.97%: 2/Feb/02 04:35: 66.77.73.77
282: 282: 7.21%: 2/Feb/02 13:08: 66.77.73.78
270: 270: 6.92%: 2/Feb/02 03:41: 66.77.73.80
272: 272: 6.97%: 2/Feb/02 03:33: 66.77.73.81
270: 270: 6.92%: 2/Feb/02 04:25: 66.77.73.82
283: 283: 7.05%: 2/Feb/02 05:40: 66.77.73.83
38: 38: 0.74%: 2/Feb/02 07:12: 66.77.73.84
272: 272: 6.97%: 2/Feb/02 06:28: 66.77.73.85
272: 272: 6.97%: 2/Feb/02 04:33: 66.77.73.86
272: 272: 6.97%: 2/Feb/02 04:16: 66.77.73.87

Go60Guy

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 408 posted 2:52 pm on Feb 6, 2002 (gmt 0)

Fast spent over four hours on my site last night. Visited hundreds of pages. Wish it could produce some significant traffic.

jimmykav

10+ Year Member



 
Msg#: 408 posted 11:46 am on Feb 8, 2002 (gmt 0)

FAST took 104 pages last night....

stephen baker

10+ Year Member



 
Msg#: 408 posted 6:35 pm on Feb 8, 2002 (gmt 0)

Hi, this is Stephen from FAST.

One of our engineers was testing the performance of a new feature using a
test-cluster of 16 crawler machines. Unfortunately, he forgot to configure
the crawlers to divide the work between them, so they all acted independetly
and effectively all crawler were crawling all webservers. We got some
feedback on this quite quickly and the configuration error was then
corrected.

Sorry for the convenience and thanks for the feedback!

jimmykav

10+ Year Member



 
Msg#: 408 posted 7:55 pm on Feb 8, 2002 (gmt 0)

Welcome Stephen
Thanks for keeping us informed.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved