Forum Moderators: open
I get spidered by these guys a lot and all I
can tell from visiting their website is that
they are a fiber network or something...
...or maybe they just have some bad spider management going on...
...or maybe they need to spider me daily
and are an information provider for some bigger entity, like AOL,
Altavista, Google, the mafia, the FBI, or
some other big company?
Does anyone know anything about this company?
Thx,
Jim
Cogent Communications Inc
But that is only my opinion.
Like they are the ISP for the Hells Angels or something???
Thx,
Jim
It looks like Metacarta could be using
66.28.68.234
66.28.68.235
66.28.68.236
66.28.68.237
to map the Web for the NSA or CIA, according to that NYT piece and our logs.
I have seen reports that User_agent ia_archiver (Wayback Machine) was appearing within a couple of small blocks inside Cogent's Class B, but this could be a cover. At any rate, Cogent sure doesn't volunteer much using various tracing tools. We have no patience with any of this stuff, so we blocked the entire Class B.
66.28.68.236
66.28.68.237
The user agent is: larbin samualt9@bigfoot.com.
jrota, do you have a user agent you can give us for your visits?
User-agent - hmmm... how are you getting an email address from these guys...is that information somewhere in the log, or are you doing a whois on the address and finding it that way?
I'll have to check the log the next time I see one, but it's definitely the 66.28.*.* net address that I remember seeing like 12 times daily...
How do I find out the User-Agent's email address?
Thx,
Jim
First of all, larbin is a French off-the-shelf bot something like Wget.
Secondly, bigfoot is now in the philippines. That email address is about as informative as a hotmail email address would be.
Finally, look at the last two hops on this trace:
19 65 ms 70 ms 71 ms metacarta.demarc.cogentco.com
20 70 ms 66 ms 68 ms 66.28.68.237
Not that their data wouldn't be interesting for the spooks... ;)
They specialize in multi-tenant office buildings, and supplying high bandwidth to the building at bargain-rate prices. Their prices for big bandwidth are so low that everyone is amazed.
Their billing is very simple and streamlined. They probably don't even know much about those who lease their bandwidth.
Many bots might be tempted to lease bandwidth from a company like this. Unfortunately, it's not easy to trace anything through Cogent because the record-keeping seems sparse.
Perhaps we should make a distinction between Cogent and those various crawlers who might be leasing bandwidth from Cogent.
...maybe someone needs to tell them they are creating a disturbance...
...except they are really good at keeping themselves hidden from enquiries, so they seem pretty good at keeping us all in suspense, wondering, nonetheless...
Jim Rota
<snip>No sigs please</snip>
[edited by: NFFC at 9:00 pm (utc) on July 30, 2002]
They ought to insist on accurate user-agents and full traceability and email addresses for reporting abuse.
On Cogent's configuration of various blocks, the tracing should provide the usual amount of information on who is assigned what blocks.
The IP numbers should all reverse-resolve.
Cogent is committing slow suicide by not insisting on this. I think it's an oversight, not a marketing plan. There's nothing more efficient than a secretive spider for generating bad publicity and paranoia.
We'll keep them blocked, thank you.
I agree that Cogent shares the blame. Anyone who sells bandwidth in large enough chunks to interest professional spiders, ought to have rules for those spiders.
I agree they should be held accountable. All too often mischievous operations are implemented behind the veil of an ISP. Of course the ISP will claim no knowledge and state that "appropriate action has been taken", when in reality they are working together.
