Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Fast Spider Details & Rundown



7:58 am on Mar 19, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Spider Type: Full multimedia Crawler (The crawler will fetch HTML documents, pictures, video, and audio.)
Robot Exclusion?: Yes, both robots.txt and meta tag noindex.
Spider Class:Agressive. (although it is much better than it used to be)
Claimed Max Pull Rate: 60seconds per html and possibly as little as 5seconds for virtual hosting on same ip. (3rd level domains can take a beating)
Spider Depth:Deep, follows all links
Fast Crawler FAQ:[fast.no]

Fast Spider IP's:
fulllist: [searchengineworld.com]

Fast spiders out of 2 partial c-blocks: exodus also owns some in the 0-127 range, although we don't think they are used for Fast.
209.202.148.* (although, up is exodus.net - which provides some net access for fast and lycos)

Agent names:

The "preXX" can be any number from 1 to 51. Some of the more popular ones:
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre18 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre19 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre20 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre24 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre25 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre26 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre27 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre30 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre34 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre40 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre41 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; [fast.no...]

There have also been sporadic hits from:
FAST-WebCrawler/3.0 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/3.0-pre1 (crawler@fast.no; [fast.no...]

Some of the more "fun" host names: admin1.bos2.fastsearch.net admin2.bos2.fastsearch.net

Their ftp/mp3 crawler: mp3.bos2.fastsearch.net mp3-1.bos2.fastsearch.net mp3-2.bos2.fastsearch.net

I assume humans (?): laptop.bos2.fastsearch.net windoze.bos2.fastsearch.net

Name servers: ns1.bos2.fastsearch.net ns2.bos2.fastsearch.net

Most Annoying Fast Fakers(?) these are rather interesting (any comments?):
¦¦FastCrawler 3.0 (crawler@1klik.dk)
¦¦FastCrawler 3.0 (crawler@1klik.dk)
ns2.jixnet.dk¦¦FastCrawler 3.0 (crawler@1klik.dk)


8:12 am on Mar 19, 2001 (gmt 0)

WebmasterWorld Administrator 10+ Year Member

I'll take that one - It's from a danish SE called 1klik [1klik.dk] It is favoring danish ".dk" domains. It is listing sites with english language though - but only on .dk domains. They are both directory and crawler based.

They are barely on the map and I'm receiving cero traffic from them. They manage a daily newsletter covering a whole lot of topics - nothing interesting to me anyway.

As a dane I'm not bothered with them - and it is sure ass h*ll not FAST engine.

Maybe one sould let them know that their spider name has a brother a little bigger ;)


2:13 pm on Mar 23, 2001 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Thanks Rambus. I just found it strange they would use that spider agent name and have nothing to do with them.

Featured Threads

Hot Threads This Week

Hot Threads This Month