homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

Fast Spider Details & Rundown

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 156 posted 7:58 am on Mar 19, 2001 (gmt 0)

Spider Type: Full multimedia Crawler (The crawler will fetch HTML documents, pictures, video, and audio.)
Robot Exclusion?: Yes, both robots.txt and meta tag noindex.
Spider Class:Agressive. (although it is much better than it used to be)
Claimed Max Pull Rate: 60seconds per html and possibly as little as 5seconds for virtual hosting on same ip. (3rd level domains can take a beating)
Spider Depth:Deep, follows all links
Fast Crawler FAQ:[fast.no]

Fast Spider IP's:
fulllist: [searchengineworld.com]

Fast spiders out of 2 partial c-blocks: exodus also owns some in the 0-127 range, although we don't think they are used for Fast.
209.202.148.* (although, up is exodus.net - which provides some net access for fast and lycos)

Agent names:

The "preXX" can be any number from 1 to 51. Some of the more popular ones:
FAST-WebCrawler/2.1-pre10 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre6 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre7 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre1 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre11 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre18 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre19 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre2 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre20 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre24 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre25 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre26 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre27 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre3 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre30 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre34 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre4 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre40 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre41 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre5 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre8 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.2-pre9 (crawler@fast.no; [fast.no...]

There have also been sporadic hits from:
FAST-WebCrawler/3.0 (crawler@fast.no; [fast.no...]
FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; [fast.no...]
FAST-WebCrawler/3.0-pre1 (crawler@fast.no; [fast.no...]

Some of the more "fun" host names: admin1.bos2.fastsearch.net admin2.bos2.fastsearch.net

Their ftp/mp3 crawler: mp3.bos2.fastsearch.net mp3-1.bos2.fastsearch.net mp3-2.bos2.fastsearch.net

I assume humans (?): laptop.bos2.fastsearch.net windoze.bos2.fastsearch.net

Name servers: ns1.bos2.fastsearch.net ns2.bos2.fastsearch.net

Most Annoying Fast Fakers(?) these are rather interesting (any comments?):
¦¦FastCrawler 3.0 (crawler@1klik.dk)
¦¦FastCrawler 3.0 (crawler@1klik.dk)
ns2.jixnet.dk¦¦FastCrawler 3.0 (crawler@1klik.dk)



WebmasterWorld Administrator 10+ Year Member

Msg#: 156 posted 8:12 am on Mar 19, 2001 (gmt 0)

I'll take that one - It's from a danish SE called 1klik [1klik.dk] It is favoring danish ".dk" domains. It is listing sites with english language though - but only on .dk domains. They are both directory and crawler based.

They are barely on the map and I'm receiving cero traffic from them. They manage a daily newsletter covering a whole lot of topics - nothing interesting to me anyway.

As a dane I'm not bothered with them - and it is sure ass h*ll not FAST engine.

Maybe one sould let them know that their spider name has a brother a little bigger ;)


WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 156 posted 2:13 pm on Mar 23, 2001 (gmt 0)

Thanks Rambus. I just found it strange they would use that spider agent name and have nothing to do with them.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved