Fast Spider Details & Rundown

[Brett_Tabke]

Spider Type: Full multimedia Crawler (The crawler will fetch HTML documents, pictures, video, and audio.)
Robot Exclusion?: Yes, both robots.txt and meta tag noindex.
Spider Class:Agressive. (although it is much better than it used to be)
Claimed Max Pull Rate: 60seconds per html and possibly as little as 5seconds for virtual hosting on same ip. (3rd level domains can take a beating)
Spider Depth:Deep, follows all links
Fast Crawler FAQ:[fast.no]

Fast Spider IP's:
fulllist: [searchengineworld.com]

Fast spiders out of 2 partial c-blocks:
209.67.247.129- 209.67.247.255 exodus also owns some in the 0-127 range, although we don't think they are used for Fast.
209.202.148.* (although, 209.202.148.128 up is exodus.net - which provides some net access for fast and lycos)

Agent names:

The "preXX" can be any number from 1 to 51. Some of the more popular ones:
FAST-WebCrawler/2.1-pre10 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.1-pre11 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.1-pre5 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.1-pre6 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.1-pre7 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre1 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre11 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre18 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre19 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre2 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre20 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre24 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre25 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre26 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre27 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre3 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre30 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre34 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre4 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre40 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre41 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre5 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre8 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.2-pre9 ([e-mail]; [fast.no...]

There have also been sporadic hits from:
FAST-WebCrawler/3.0 ([e-mail]; [fast.no...]
FAST-WebCrawler/2.1-pre5 ([e-mail]; [fast.no...]
FAST-WebCrawler/3.0-pre1 ([e-mail]; [fast.no...]

Some of the more "fun" host names:
209.202.148.9 admin1.bos2.fastsearch.net
209.202.148.10 admin2.bos2.fastsearch.net

Their ftp/mp3 crawler:
209.202.148.45 mp3.bos2.fastsearch.net
209.202.148.46 mp3-1.bos2.fastsearch.net
209.202.148.47 mp3-2.bos2.fastsearch.net

I assume humans (?):
209.202.148.55 laptop.bos2.fastsearch.net
209.202.148.56 windoze.bos2.fastsearch.net

Name servers:
209.202.148.91 ns1.bos2.fastsearch.net
209.202.148.92 ns2.bos2.fastsearch.net

Most Annoying Fast Fakers(?) these are rather interesting (any comments?):
¦212.97.217.26¦FastCrawler 3.0 ([e-mail])
¦212.97.217.27¦FastCrawler 3.0 ([e-mail])
ns2.jixnet.dk¦194.239.192.2¦FastCrawler 3.0 ([e-mail])

[Rumbas]

I'll take that one - It's from a danish SE called 1klik [1klik.dk] It is favoring danish ".dk" domains. It is listing sites with english language though - but only on .dk domains. They are both directory and crawler based.

They are barely on the map and I'm receiving cero traffic from them. They manage a daily newsletter covering a whole lot of topics - nothing interesting to me anyway.

As a dane I'm not bothered with them - and it is sure ass h*ll not FAST engine.

Maybe one sould let them know that their spider name has a brother a little bigger ;)

[Brett_Tabke]

Thanks Rambus. I just found it strange they would use that spider agent name and have nothing to do with them.