Welcome to WebmasterWorld Guest from 54.196.233.239

Forum Moderators: open

Message Too Old, No Replies

Fast Spider Details & Rundown

     

Brett_Tabke

7:58 am on Mar 19, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Spider Type: Full multimedia Crawler (The crawler will fetch HTML documents, pictures, video, and audio.)
Robot Exclusion?: Yes, both robots.txt and meta tag noindex.
Spider Class:Agressive. (although it is much better than it used to be)
Claimed Max Pull Rate: 60seconds per html and possibly as little as 5seconds for virtual hosting on same ip. (3rd level domains can take a beating)
Spider Depth:Deep, follows all links
Fast Crawler FAQ:[fast.no]

Fast Spider IP's:
fulllist: [searchengineworld.com]

Fast spiders out of 2 partial c-blocks:
209.67.247.129- 209.67.247.255 exodus also owns some in the 0-127 range, although we don't think they are used for Fast.
209.202.148.* (although, 209.202.148.128 up is exodus.net - which provides some net access for fast and lycos)

Agent names:

The "preXX" can be any number from 1 to 51. Some of the more popular ones:
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre18 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre19 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre20 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre24 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre25 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre26 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre27 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre30 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre34 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre40 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre41 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; [fast.no...]

There have also been sporadic hits from:
FAST-WebCrawler/3.0 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/3.0-pre1 (crawler@fast.no; [fast.no...]

Some of the more "fun" host names:
209.202.148.9 admin1.bos2.fastsearch.net
209.202.148.10 admin2.bos2.fastsearch.net

Their ftp/mp3 crawler:
209.202.148.45 mp3.bos2.fastsearch.net
209.202.148.46 mp3-1.bos2.fastsearch.net
209.202.148.47 mp3-2.bos2.fastsearch.net

I assume humans (?):
209.202.148.55 laptop.bos2.fastsearch.net
209.202.148.56 windoze.bos2.fastsearch.net

Name servers:
209.202.148.91 ns1.bos2.fastsearch.net
209.202.148.92 ns2.bos2.fastsearch.net

Most Annoying Fast Fakers(?) these are rather interesting (any comments?):
¦212.97.217.26¦FastCrawler 3.0 (crawler@1klik.dk)
¦212.97.217.27¦FastCrawler 3.0 (crawler@1klik.dk)
ns2.jixnet.dk¦194.239.192.2¦FastCrawler 3.0 (crawler@1klik.dk)

Rumbas

8:12 am on Mar 19, 2001 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



I'll take that one - It's from a danish SE called 1klik [1klik.dk] It is favoring danish ".dk" domains. It is listing sites with english language though - but only on .dk domains. They are both directory and crawler based.

They are barely on the map and I'm receiving cero traffic from them. They manage a daily newsletter covering a whole lot of topics - nothing interesting to me anyway.

As a dane I'm not bothered with them - and it is sure ass h*ll not FAST engine.

Maybe one sould let them know that their spider name has a brother a little bigger ;)

Brett_Tabke

2:13 pm on Mar 23, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Thanks Rambus. I just found it strange they would use that spider agent name and have nothing to do with them.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month