Interesting spiders from AV & Mercator

[Air]

Anyone have any comments on the following list of IP's and User Agents, I find the Mercator/Scooter one particularly interesting. All from Nov. 11/2000 to Nov. 25/2000.

User Agent: Scooter/1.0
IP Address: 209.73.164.13
User Agent: Scooter-1.0
IP Address: 209.73.164.42
User Agent: Scooter/2.0 G.R.A.B. V1.1.0
IP Address: 209.73.164.127
User Agent: TV35_Mercator_6-1.0
IP Address: 209.73.164.129
User Agent: TV36_Mercator_11-1.0
IP Address: 209.73.164.130
User Agent: Scooter2_Mercator_3-1.0
IP Address: 209.73.164.40

[PeteU]

yeah, I've been watching them last couple weeks, first came Scooter/2.0 G.R.A.B. V1.1.0, then Mercator-1.0, after a while seems they had babies and Scooter2_Mercator_4-1.0 varieties started showing up and finally TV's started showing just today, there may be some meaning to all this...
Also, there are now different agents from the same IP like:
TV35_Mercator_6-1.0
TV35_Mercator_7-1.0
TV35_Mercator_4444-1.0
all from 209.73.164.129
Maybe Alta will now start to use Mercator data in some new
algorithm, possibly what was in white papers discussed in another thread, and this is part of a spidering for new database. Hopefully something usefull will come out of this, so far Mercator has just been a bandwidth waster.

[Air]

>Maybe Alta will now start to use Mercator data in some new
algorithm

yeah, that's what I was thinking Pete, the from field has an AV email addy, and the IP's resolve to hostnames like:

tv35.sv.av.com
scooter.sv.av.com (the Scooter2_Mercator combo)

which all make me think it even more ...

[littleman]

I've also been watching them. The pattern on my domains has been the same as what you outlined Peter. I wonder if we are going to see some fundamental changes in the db algorithm soon?

[Cisco]

I don't know what's up with this new Scooter2-Mercator combo, but I can tell you that it seems to totally disregard the robots.txt file.

Good thing I password protect my real (cloaked) pages dir.

[msgraph]

We are receiving a large amount of spidering from Altavista's Mercator robots.(In the tens of thousands) These look like new versions from the looks of the various user agent names. They all come off of the same C-block of a Scooter/2.0 robot I found last week so I don't know what's going on.

User Agents:

TV36_Mercator_10-1.0
TV35_Mercator_4444-1.0
TV34_Mercator_fxa-1.0
TV33_Mercator_2-1.0

IP Range: 209.73.164.40 - 209.73.164.126

Now I am getting this User agent coming from the IP address listed below it.

User agent: Scooter2_Mercator_6-1.0
IP address: 209.73.164.42

And guess what? This exact IP had a different User agent name last week. It was Scooter/2.0 G.R.A.B. V1.1.0

I did a DNS lookup on this address and it returned as test-scooter.sv.av.com.

I'm beginning to think that these Mercator agents are in fact Altavista's own doing. They must be test robots used to work out some bugs before they actually become a Scooter version. Everyone around the net is saying that it is just a research spider running off their computers with no connection but maybe there is more to it than that.

Altavista will never reply on my questions about Mercator.(Not that they reply to anything anyway!!!)

[oLeon]

Yes,
these are definitive from Altavista, because they
-that means
#UA Scooter_Mercator_1-1.0 = IP 209.73.164.40
#UA TV36_Mercator_8-1.0 = IP 209.73.164.130 -
spidered pages which were only submitted to Altavista.com

I got some more informations about them and their brothers and sisters:

#UA TV33_Mercator_1-1.0
209.73.164.126
#UA TV34_Mercator_4444-1.0
209.73.164.127
#UA TV35_Mercator_4444-1.0
209.73.164.129
#UA Scooter2_Mercator_4-1.0
209.73.164.40

[cirelle]

anybody have any ideas about this group of Mercator's

Mercator-2.0
204.123.28.30
204.123.28.31
204.123.28.32
204.123.28.33

Digital Equipment Corporation (NETBLK-DEC-P)
Digital Equipment Corporation
Network Systems Laboratory
250 University Avenue
Palo Alto, CA 94301-1616

Netname: DEC-P
Netblock: 204.123.0.0 - 204.123.255.0

Coordinator:
Saurus, Skip (SS486-ARIN) [e-mail]
[phone]

maybe the names have'nt been changed yet?

c

[msgraph]

I've tried to contact Altavista many times about Mercator and I get the same response each time.

"Yes indeed they are running off our spiders"

[Brett_Tabke]

<aol>me3</aol>

209.73.164.129
tv35.sv.av.com
Scooter2_Mercator_url-1.0

209.73.164.40
scooter.sv.av.com
Scooter2_Mercator_3-1.0

209.73.164.41
scooter2.sv.av.com
Scooter2_Mercator_lca-1.0

In a related note, there was a blurb on the pr wire earlier in the week about Alta releasing an upgraded version of its se software. Maybe they are just out trolling around testing it?

[EliteWeb]

Mercator-2.0 was all over my site today. 638 times within a few hours.

[msgraph]

Is the IP from Alta?

[EliteWeb]

Name: atrax0.pa-x.dec.com
Address: 204.123.28.30

[wilderness]

I had this same Mercator 2.0 from dec.com. Same IP as well. Today.
It only read the robots and one other file.
I looked all over the Compaq/DEC website for a reference to a bot with no success.
Emailed DEC with no reply so far.

[msgraph]

Here you go wilderness

[webmasterworld.com...]

Interesting note is that one of the creators, Marc Najork, works for MS research now. BUT don't get your hopes up on Microsoft. He's doing 3D type work I think.

[research.microsoft.com...]

(edited by: msgraph at 12:48 am (utc) on May 17, 2002)

[wilderness]

Many thanks msgraph.
Heavy duty reading.

Check out this link from Marc's page if you have a DSL or CABLE
[research.microsoft.com...]

[msgraph]

Mercator is grabbing images as well. Anyone else seeing this?

Also, it appears that if you call an image on one site(A) from another site(B), without a link tag, Mercator will crawl site B.