Welcome to WebmasterWorld Guest from 50.19.0.90

Forum Moderators: open

Message Too Old, No Replies

Fast Spider Details & Rundown

     
7:58 am on Mar 19, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Spider Type: Full multimedia Crawler (The crawler will fetch HTML documents, pictures, video, and audio.)
Robot Exclusion?: Yes, both robots.txt and meta tag noindex.
Spider Class:Agressive. (although it is much better than it used to be)
Claimed Max Pull Rate: 60seconds per html and possibly as little as 5seconds for virtual hosting on same ip. (3rd level domains can take a beating)
Spider Depth:Deep, follows all links
Fast Crawler FAQ:[fast.no]

Fast Spider IP's:
fulllist: [searchengineworld.com]

Fast spiders out of 2 partial c-blocks:
209.67.247.129- 209.67.247.255 exodus also owns some in the 0-127 range, although we don't think they are used for Fast.
209.202.148.* (although, 209.202.148.128 up is exodus.net - which provides some net access for fast and lycos)

Agent names:

The "preXX" can be any number from 1 to 51. Some of the more popular ones:
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre18 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre19 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre20 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre24 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre25 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre26 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre27 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre30 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre34 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre40 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre41 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; [fast.no...]

There have also been sporadic hits from:
FAST-WebCrawler/3.0 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/3.0-pre1 (crawler@fast.no; [fast.no...]

Some of the more "fun" host names:
209.202.148.9 admin1.bos2.fastsearch.net
209.202.148.10 admin2.bos2.fastsearch.net

Their ftp/mp3 crawler:
209.202.148.45 mp3.bos2.fastsearch.net
209.202.148.46 mp3-1.bos2.fastsearch.net
209.202.148.47 mp3-2.bos2.fastsearch.net

I assume humans (?):
209.202.148.55 laptop.bos2.fastsearch.net
209.202.148.56 windoze.bos2.fastsearch.net

Name servers:
209.202.148.91 ns1.bos2.fastsearch.net
209.202.148.92 ns2.bos2.fastsearch.net

Most Annoying Fast Fakers(?) these are rather interesting (any comments?):
¦212.97.217.26¦FastCrawler 3.0 (crawler@1klik.dk)
¦212.97.217.27¦FastCrawler 3.0 (crawler@1klik.dk)
ns2.jixnet.dk¦194.239.192.2¦FastCrawler 3.0 (crawler@1klik.dk)

8:12 am on Mar 19, 2001 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
posts:2530
votes: 1


I'll take that one - It's from a danish SE called 1klik [1klik.dk] It is favoring danish ".dk" domains. It is listing sites with english language though - but only on .dk domains. They are both directory and crawler based.

They are barely on the map and I'm receiving cero traffic from them. They manage a daily newsletter covering a whole lot of topics - nothing interesting to me anyway.

As a dane I'm not bothered with them - and it is sure ass h*ll not FAST engine.

Maybe one sould let them know that their spider name has a brother a little bigger ;)

2:13 pm on Mar 23, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Thanks Rambus. I just found it strange they would use that spider agent name and have nothing to do with them.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members