homepage Welcome to WebmasterWorld Guest from 54.161.236.229
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
Fast Spider Details & Rundown
Brett_Tabke




msg:222413
 7:58 am on Mar 19, 2001 (gmt 0)

Spider Type: Full multimedia Crawler (The crawler will fetch HTML documents, pictures, video, and audio.)
Robot Exclusion?: Yes, both robots.txt and meta tag noindex.
Spider Class:Agressive. (although it is much better than it used to be)
Claimed Max Pull Rate: 60seconds per html and possibly as little as 5seconds for virtual hosting on same ip. (3rd level domains can take a beating)
Spider Depth:Deep, follows all links
Fast Crawler FAQ:[fast.no]

Fast Spider IP's:
fulllist: [searchengineworld.com]

Fast spiders out of 2 partial c-blocks:
209.67.247.129- 209.67.247.255 exodus also owns some in the 0-127 range, although we don't think they are used for Fast.
209.202.148.* (although, 209.202.148.128 up is exodus.net - which provides some net access for fast and lycos)

Agent names:

The "preXX" can be any number from 1 to 51. Some of the more popular ones:
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre18 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre19 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre20 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre24 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre25 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre26 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre27 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre30 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre34 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre40 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre41 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; [fast.no...]

There have also been sporadic hits from:
FAST-WebCrawler/3.0 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/3.0-pre1 (crawler@fast.no; [fast.no...]

Some of the more "fun" host names:
209.202.148.9 admin1.bos2.fastsearch.net
209.202.148.10 admin2.bos2.fastsearch.net

Their ftp/mp3 crawler:
209.202.148.45 mp3.bos2.fastsearch.net
209.202.148.46 mp3-1.bos2.fastsearch.net
209.202.148.47 mp3-2.bos2.fastsearch.net

I assume humans (?):
209.202.148.55 laptop.bos2.fastsearch.net
209.202.148.56 windoze.bos2.fastsearch.net

Name servers:
209.202.148.91 ns1.bos2.fastsearch.net
209.202.148.92 ns2.bos2.fastsearch.net

Most Annoying Fast Fakers(?) these are rather interesting (any comments?):
¦212.97.217.26¦FastCrawler 3.0 (crawler@1klik.dk)
¦212.97.217.27¦FastCrawler 3.0 (crawler@1klik.dk)
ns2.jixnet.dk¦194.239.192.2¦FastCrawler 3.0 (crawler@1klik.dk)

 

Rumbas




msg:222414
 8:12 am on Mar 19, 2001 (gmt 0)

I'll take that one - It's from a danish SE called 1klik [1klik.dk] It is favoring danish ".dk" domains. It is listing sites with english language though - but only on .dk domains. They are both directory and crawler based.

They are barely on the map and I'm receiving cero traffic from them. They manage a daily newsletter covering a whole lot of topics - nothing interesting to me anyway.

As a dane I'm not bothered with them - and it is sure ass h*ll not FAST engine.

Maybe one sould let them know that their spider name has a brother a little bigger ;)

Brett_Tabke




msg:222415
 2:13 pm on Mar 23, 2001 (gmt 0)

Thanks Rambus. I just found it strange they would use that spider agent name and have nothing to do with them.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved