flunky -- anyone seen this spider?

66.28.20.194 flunky -- crawling my sites.

[gethan]

At the rate of about one page per hour ;)

Never seen or heard of it before... picks up my robots.txt, and also looks at my php pages.

Anyone know anything about it?

My searches didn't reveal anything ...

ps. hope this is a good forum for this.

[IanTurner]

I'd try spider identification forum, but I haven't seen flunky

[ggrot]

I saw it too - just today. No insight on it though. The site it spidered isn't very high profile, so it is probably bigger than some individual on a cable modem.

[gethan]

[Originally posted in Alternative Search Engines]
[webmasterworld.com...]

Flunky has been crawling my site at the rate of about one page per hour. Never seen or heard of it before... picks up my robots.txt, and also looks at my php pages.

Anyone know anything about it?

My searches didn't reveal anything ...

From ggrot -

I saw it too - just today. No insight on it though. The site it spidered isn't very high profile, so it is probably bigger than some individual on a cable modem.

[gethan]

Ripe whois gives me

66.28.20

% This is the RIPE Whois server.
% The objects are in RPSL format.
% Please visit [ripe.net...] for more information.
% Rights restricted by copyright.
% See [ripe.net...]

inetnum: 0.0.0.0 - 255.255.255.255
netname: IANA-BLK
descr: The whole IPv4 address space
country: NL
admin-c: IANA1-RIPE
tech-c: IANA1-RIPE
status: ALLOCATED UNSPECIFIED

To me that reads as -- "ripe doesn't know".

[Josk]

Thats interesting...I saw this guy on 16/7/01 and 17/7/01, but then it was as "larbin_2.1.1 [e-mail]".

Now it comes back as "flunky", and to the same server on 16/9/01.

That was some summer holiday it had... Must have been a life changing experience to go change its name like that!

[gethan]

duh ... ripe == europe ;)

So I tried the US equiv.

[arin.net...]

Cogent Communications (NETBLK-COGENT-NB-0000)
1015 31st Street, NW
Washington, DC 20007
US

Netname: COGENT-NB-0000
Netblock: 66.28.0.0 - 66.28.127.255
Maintainer: COGC

Coordinator:
Cogent Communications (ZC108-ARIN) [e-mail]
[phone]

Domain System inverse mapping provided by:

AUTH1.DNS.COGENTCO.COM66.28.0.14
AUTH2.DNS.COGENTCO.COM66.28.0.30

ADDRESSES WITHIN THIS BLOCK ARE NON-PORTABLE

Record last updated on 22-Aug-2001.
Database last updated on 17-Sep-2001 23:12:41 EDT.

So what are Cogent up to?

(I think Cogent are some miltary communications company related to Nortel ... talks about optical on the website???)

[edited -- wrong ip submitted to arin -- * slaps forehead *]

[volatilegx]

Cogent is a fiberoptic cable company offering super high speed access to corporations, etc.

I've seen flunky come from the following IPs recently:

66.28.20.194
66.28.44.122
66.28.44.123
66.28.44.125

[IanTurner]

I am also now seeing flunky in the logs

[littleman]

66.28.44.121 is a password protected site -> They named the server temple.metacarta.com, which is a bad thing for them to do if they wanted to cover their tracks...
It is running Apache/1.3.19 (Unix) Debian/GNU/

From metacarta.com:

Company Profile

MetaCarta maps information to the physical world, increasing the value of an organization's knowledge by leveraging existing data. In a world of global competition, understanding complex relationships determines success. MetaCarta digests massive amounts of information in a novel way, creating a cohesive picture to accelerate decision making.

blahhblahhblahh...

If you do a traceroute on any of the .12* ips the last hop is 66.28.20.194, which has the same type of authentication message -> "Pleas contact jrf with questions."

[Great1]

Yeah, flunky hit my hard yesterday. Spidered constantly for about 9 hours. The 1st to find out why flunky is spidering gets a lolly.

[volatilegx]

flunky looks like the perfect candidate for banning.

[gethan]

>flunky looks like the perfect candidate for banning.

Its fairly low impact - (on my site) I think I'll leave banning it just in case it turns out to be something cool.

MetaCarta maps information to the physical world, increasing the value of an organization's knowledge by leveraging existing data. In a world of global competition, understanding complex relationships determines success. MetaCarta digests massive amounts of information in a novel way, creating a cohesive picture to accelerate decision making.

Translated:

We steal your content and sell it.

?

[Jaf]

Looks like flunky now has some brothers and sisters. On my site I originally saw flunky come from IP address 66.28.44.125, and today from addresses 66.28.44.122-124 I've seen

RRC ([e-mail])

This means nothing to me (I guess we could mail them :-). It only hit three pages, so that's pretty small beer for now.

[littleman]

More bigfoot IPs:
66.28.68.234-237

[Everyman]

Some of our domains, which have political content, are getting hit hard by Metacarta. The IP never reverse-resolves. Here are the ones we believe are Metacarta:

66.28.68.234
66.28.68.235
66.28.68.236
66.28.68.237
66.28.250.171
66.28.250.172
66.28.250.173
66.28.250.174

Some substantiation comes from the last two hops on a traceroute:

19 65 ms 70 ms 71 ms metacarta.demarc.cogentco.com
20 70 ms 66 ms 68 ms 66.28.68.237

As reported in the New York Times [nytimes.com] last January, Metacarta is keen on getting a spook contract from the feds for Internet snooping and mapping.

Contrary to some of the posts above, I don't see any relation between Metacarta and bigfoot.com. The former is in Massachusetts, the latter in the Philippines. While Metacarta has a website that says almost nothing, if you look under "jobs" you will see that they want a sales engineer with a security clearance to work out of Washington DC (where cogentco.com is located).

I suspect that they got their spook contract. Does anyone have any other IP numbers for Metacarta crawlers?