Scooter-3.2.DIL

[Crazy_Fool]

64.152.75.114 Scooter-3.2.DIL
hit robots.txt and index.html twice each, came back 10 minutes later and hit them both again. then came back 1h 50mins later and ripped through 58 page site in 1min 50seconds. 8 hours later it did exactly the same again with approximately the same time intervals.

[Air]

Scooter has been tearing through a few sites, they have been the greediest spider in my logs for a couple months now. It wouldn't be so bad if it actually led to something ...

[Crazy_Fool]

i'd be happy if they just listed my sites somewhere in the top 10,000 pages ....

anyone got any idea why AV feels the need to create new spiders every month? i haven't been monitoring the AV UAs and IPs, but are new UAs coming out on existing IPs or are they new IPs each month? do all the new UAs we've seen keep on going month after month or are they replaced every month or so?

[volatilegx]

I've seen a big increase lately in referrals from AV. I don't know whether this is from an improvement in rankings of old pages, recent indexing of pages submitted long ago, or just an increase of traffic at AV. Anyhoo, AV is finally sending traffic my way again.

[wilderness]

Perhaps I'm dense :-)

The past few days I been made to wonder if ALL these other Scooters are part of Altavista?

The AV Scooter
AltaVista Company (NETBLK-INTERNET-BLK-1-AV)
529 Bryant St.
Palo Alto, CA 94301
US

Netname: INTERNET-BLK-1-AV
Netblock: 209.73.160.0 - 209.73.191.255

The OTHER Scooters ALL come from The Level 3

Level 3 Communications, Inc. (NET-LEVEL5-CIDR) LEVEL5-CIDR
166.90.0.0 - 166.90.255.255
Level 3 Communications, Inc. (NET-LEVEL6-CIDR) LEVEL6-CIDR
209.0.0.0 - 209.0.255.255
Level 3 Communications, Inc. (NET-LEVEL7-CIDR) LEVEL7-CIDR
207.90.128.0 - 207.90.191.255
Level 3 Communications, Inc. (NET-LEVEL8-CIDR) LEVEL8-CIDR
206.15.0.0 - 206.15.31.255
Level 3 Communications, Inc. (NET-LEVEL9-CIDR) LEVEL9-CIDR
192.231.42.0 - 192.231.42.255
Level 3 Communications, Inc. (NETBLK-LC-ORG-ARIN) LC-ORG-ARIN
64.152.0.0 - 64.159.255.255
Level 3 Communications, Inc. (NETBLK-LEVEL3-CIDR) LEVEL3-CIDR
209.244.0.0 - 209.247.255.255
Level 3 Communications, Inc. (NETBLK-LEVEL4-CIDR) LEVEL4-CIDR
63.208.0.0 - 63.215.255.255

end of quote.

NOWHERE is Level 3's IP ranges do I see
209.73.162.133

[wilderness]

<snip>Is anyone else beginning to feel the need to block Scooter?
He's been at my site requesting pages that haven't existed for months. But he keeps insisting that they are there and continues to check. >

mdharrold,
I blocked the Level 3 Scooters the other day when I submitted the above.

The only one I allow is the AV Scooter
209.73.162.

I am getting a rash of referrals from UK AV which I really not interested in having.

My US AV referrals are a small percentage of my visitors.

[Key_Master]

My problem with Scooter-3.2.DIL is that sometimes it tries to hit pages with .htm extensions. The pages it is looking for exist on my site but I use .html extensions. It's creating a lot of 404 errors.

Very annoying. :(

[David]

>> My problem with Scooter-3.2.DIL is that sometimes it tries to hit pages with .htm extensions. <<

I am having the same problem. This is the first time It has happened to me.

Does any one know why Scooter is asking for an extension that does not exist?

[JuniorHarris]

From what I noticed, it appeared as if it was simply dropping off the last character.

This wouldn't be the first time they released a buggy spider!~

Now if AV could send at least half as many referrals as we get spider hits!~

[Key_Master]

Here's what AltaVista had to say about it:

Thanks for contacting us about the machine 64.152.75.44
(scooter13.sv.av.com). This machine is running a web crawler on behalf of
AltaVista, and is probably following links that it found on some other page.
If your machine is not really a web server, or if you have blocked access to
it from the outside, our crawler will try only a few times before giving up.

Personally, I would like to know whose page is linking to the following path/file on my site:

/kjhgdkjhf1goifj2lktjelj34knfhjguih8bbj/index.html

Needless to say, it doesn't exist on my site. And even if it did, I doubt I'd be able to remember where it was. ;)

[wilderness]

<snip>or if you have blocked access to it from the outside, our crawler will try only a few times before giving up.>

Obvioulsy the AV bots follow predetermined policy
:-) Tongue-in-cheek

This bot was denied for over a week and NEVER left or gave up.
On Saturday I removed the deny and added a disallow to my robots which AV conformed with.
However almost instantly another Scooter appeared from the 209.73.162. block :-)

[TallTroll]

Is anyone else seeing the 3.3dev spider out of 209.73.174.251?

[HandwovenRug]

I had hits from 209.73.174.251, that's avfw.sv.av.com. Here's the log:

209.73.174.251 - - [15/Feb/2002:12:00:20 +0000] "GET /robots.txt HTTP/1.1" 206 265 "-" "Scooter-3.3dev"

[wilderness]

Had a AV hit today using a Java as UA.
Didn't even read the robots :-(

209.73.164.50 - - [22/Feb/2002:18:10:35 -0800] "HEAD /BettinS/Books.htm HTTP/1.1" 200 0 "-" "Java1.3.1"

[JuniorHarris]

Dont' think that is a spider, looks like the address from av's translator [babelfish.altavista.com].