Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast

Something very fishy about this one



9:37 pm on Mar 3, 2006 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member - - [03/Mar/2006:04:23:15 -0600] "GET /some_page.html HTTP/1.1" 403 1492 "-" "FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast"

The IP address resolves only as far as HostForWeb Inc, in Chicago, Il. Digging deeper, you get only

Asking ns1.scservers.com. for PTR record: Reports unknown.ord.scnet.net. [from]
where scnet.net no longer resolves.

As I understood/remember it, Fast divested AllTheWeb to concentrate on enterprise search solutions a few years ago. And I'm not aware of any relationship at all between Lycos in MA, and any other company in IL.

Since it didn't fetch robots.txt, it didn't get anywhere, but keep an eye out for this one.

If it's legitimate, then it needs two emergency repairs: Fetch and obey robots.txt, and provide valid contact info in the UA string.



6:52 am on Mar 4, 2006 (gmt 0)

5+ Year Member

I've been seeing this bot in my logs for a little while now. At first it came with a user agent of just random letters such as:l2axlqwvk2bguksmie+vbsvgaomkxbl - -

Then it started using this:
FAST-WebCrawler/2.2.5+-+Lycos/Alltheweb/Fast - -

So far it has come from:


It mostly uses the Fast UA now.

I can't confirm any of these IPs as Fast

I've blocked them all by IP since the random UA can't be used.

Anyone else seen these IP or UAs?


8:15 pm on Mar 5, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

I had a visit from this one last week. It did not request robots.txt and quickly fell into a spider trap. Sticky me if you want the last octet of the IP Address.

FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast

Partial WHOIS:

OrgName: Columbus Network Access Point, Inc.
Address: 50 W, Broad St, Suite 627
City: Columbus
StateProv: OH
PostalCode: 43215
Country: US
NetRange: -
NetHandle: NET-209-190-0-0-1
Parent: NET-209-0-0-0-0
NetType: Direct Allocation
RegDate: 1997-12-19
Updated: 2005-03-29


9:58 pm on Mar 5, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member

FAST-related bots have been a plague (IMHO) for years. If I even see the word "FAST" in my logs, I practically start to twitch:)

Here's a mini assortment of UAs from my robots.txt, not that FAST reliably heeds them:

User-agent: FAST
User-agent: FAST Enterprise Crawler
User-agent: FAST-WebCrawler
User-agent: FAST MetaWeb Crawler
Disallow: /

Here are some older hit/hosts:

cr022r01-2.sac2.fastsearch.net - - [13/Oct/2002:11:20:21 -0700]
"FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

cr022r01-3.sac2.fastsearch.net - - [21/Jan/2004:10:40:32 -0800]
"FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; [alltheweb.com...]

cr022r01-3.sac.overture.com - - [04/Apr/2004:15:06:38 -0700]
"FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; [alltheweb.com...]

And here are a couple of the newest:

sch-fast-se-crawl01.dev.osl.basefarm.net - - [01/Mar/2006:00:58:31 -0800]
"GET /robots.txt HTTP/1.1"
"FAST Enterprise Crawler 6 used by Schibsted Sok (webcrawl@schibstedsok.no)" - - [01/Mar/2006:08:31:28 -0800]
"GET /robots.txt HTTP/1.1"
"FAST Enterprise Crawler 6 used by FAST (iverjor (at) fast.no)"
(Eight minutes later, this one hit my homepage. Grrr...)

Over the years, FAST IPs have tracked back to Norway, and Massachusetts, as I recall, and goodness knows where else. This time around, "" hails from Tokyo, Japan.

Nowadays, the minute I see a FAST IP, if it's getting 403'd for not reading/heeding robots.txt, I block it in the firewall. An overreaction, perhaps, but too many FAST-running individuals/companies have scraped the paint off the walls too many times.


10:03 pm on Mar 5, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member


On a related note, here's one more:

dnaspider04.mia.lycos.com - - [04/Feb/2006:18:16:02 -0800]
"GET /robots.txt HTTP/1.0"


User-agent: Lycos
User-agent: Lycos_Spider_(T-Rex)
User-agent: Lycos_Spider_(modspider)


10:42 pm on Mar 5, 2006 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Fast is a legitimate search company, previously powering the "AllTheWeb" search site -- actually a very good search engine -- but having now sold it and moved back into enterprise search exclusively. The Fast requests from .se domains are likely to be legitimate.

But I suspect the user-agent in the title of this thread is a spoof.



12:37 am on Mar 6, 2006 (gmt 0)

5+ Year Member

Thanks for the info.

I also can confirm 209.190.21.*

Blocked by IP.


12:48 am on Mar 7, 2006 (gmt 0)

5+ Year Member


FAST-WebCrawler/2.2.5+-+Lycos/Alltheweb/Fast - -


This has to be a distributed spider or maybe a spammers group?


2:09 pm on Mar 7, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Fast sells enterprise searching software. Perhaps the spider is part of their deal? This spider could be coming from anywhere/everywhere.


2:27 pm on Mar 7, 2006 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Might be a 'bot sold by Fast, but...

> Since it didn't fetch robots.txt

Never had that problem with Fast themselves.



2:53 pm on Mar 15, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Here's an interesting tidbit... "FAST MetaWeb Crawler (helpdesk at fastsearch dot com)" sends an accurate HTTP_REFERER header, unlike most spiders.

The IP is registered to Fast Search and Transfer.


12:18 am on Mar 16, 2006 (gmt 0)

10+ Year Member

Saw this one a few times back in February, but the past few days it is spidering my site heavily. It now uses some weird random UA names like "pynbjmxjcStcooccwSyd", "srhpdbdsgqsygi7ctsxpmwpdhqhmgwkfieeiwsy" or "dohdegijvi3GyxnirfqGfcwjvgodgghex3jict" and many more.
I just blocked the IP range 205.234.253.*, that are the only ones I've seen. The other IP ranges mentioned before I haven't seen.


11:44 pm on Mar 19, 2006 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member

For those of you keeping score at home, here's Yet Another FAST-related bot variation/name:

schibstedsokbot (compatible; Mozilla/5.0; MSIE 5.0; FAST FreshCrawler 6; +http://www.schibstedsok.no/bot/)
03/18 20:25:59 /robots.txt 200 -

It's not consistent re robots.txt -- sometimes that's all it hits, sometimes not at all -- which is why I block all FAST spawn.

P.S. to adb64

It's been my experience that nonsensical UAs typically aren't FAST-related, and perhaps that explains why the IPs didn't match for you. I'm not sure who/what is behind the nonsense -- could be individuals playing with their browsers or a browser extension, or some program covering its tracks. (I suspect an extension or program.) Here are some similar fake UAs I've seen recently:

m bm9nswptqddtxqtrfjfqwur
kknfrskhn cxydbj9fymyhklr
rpy edmsjvblflwdx0tsromet0n0v

Those get blocked automatically, but if I see repeated hits from the same ISP, or the IP/host name tracks back to a server farm, I rewrite the host, too. FWIW


Featured Threads

Hot Threads This Week

Hot Threads This Month