Altavista's scooter

Forum Moderators: open

Message Too Old, No Replies

Altavista's scooter

msgraph

3:45 pm on Jan 30, 2001 (gmt 0)

Is everyone seeing various Scooter spiders hitting their sites today? Going after the root index and robots file? I've seen them run across a few hundred domains so far. Many of them haven't been submitted in some time either.

UA :
Scooter-tv36_Mc_f65m_32
Scooter-tv34_Mc_f65m_16

New index in the making pretty soon or just looking for updates...who knows?

Air

5:27 pm on Jan 31, 2001 (gmt 0)

Yes have seen them hit as well. There was another round of these types of User Agents from AV/Merc about 8 weeks ago. They did a pretty complete crawl, but I can't say I saw any additional pages indexed.

Here are some other threads relating to that UA:
[webmasterworld.com...]
[webmasterworld.com...]

msgraph

5:39 pm on Jan 31, 2001 (gmt 0)

Did some more checking and it is following Merc's pattern on both the IP and crawling pattern. I see the Merc 1.0 coming in today as well.

Anyone have monkeyboy in AV to provide some clues on this so-called "research spider"?

I'm really getting annoyed at them eating up my bandwidth for something they won't provide any info on.

Air

10:01 pm on Jan 31, 2001 (gmt 0)

I am assuming at the moment that since they carry a partial UA of Scooter that the data they spider must be available to AV, so I treat them as AV. I guess you could block it from spidering if you wanted to, but IMO it would be a mistake until more is known about this relationship.