Another new Scooter UA

209.73.162.151 Scooter_trk15-3.0.3

[Crazy_Fool]

Arrived on one of my sites today. 17 seconds later had ripped through every page on the site. Luckily that site was only about 25 pages. Watch out WmW!!!

[bobriggs]

Yeah, the one on my site is:

trek29.sv.av.com (209.73.162.82)

Scooter_trk28-3.0.3

[Crazy_Fool]

so ... with all these new scooters, how do we block scooter ? do we just use "Scooter" in the robots.txt file or do we need to use the full UA for every scooter agent we see ?

[littleman]

I got it coming in from 209.73.162.71.

[Josk]

So far I've seen:

+---------------------+----------------+
¦ SPIDER_UA ¦ SPIDER_IP ¦
+---------------------+----------------+
¦ Scooter_trk1-3.0.3 ¦ 209.73.162.51 ¦
¦ Scooter_trk12-3.0.3 ¦ 209.73.162.20 ¦
¦ Scooter_trk15-3.0.3 ¦ 209.73.162.151 ¦
¦ Scooter_trk16-3.0.3 ¦ 209.73.162.161 ¦
¦ Scooter_trk17-3.0.3 ¦ 209.73.162.171 ¦
¦ Scooter_trk18-3.0.3 ¦ 209.73.162.181 ¦
¦ Scooter_trk20-3.0.3 ¦ 209.73.162.2 ¦
¦ Scooter_trk21-3.0.3 ¦ 209.73.162.12 ¦
¦ Scooter_trk22-3.0.3 ¦ 209.73.162.22 ¦
¦ Scooter_trk24-3.0.3 ¦ 209.73.162.42 ¦
¦ Scooter_trk25-3.0.3 ¦ 209.73.162.52 ¦
¦ Scooter_trk4-3.0.3 ¦ 209.73.162.81 ¦
¦ Scooter_trk6-3.0.3 ¦ 209.73.162.101 ¦
¦ Scooter_trk9-3.0.3 ¦ 209.73.162.131 ¦
+---------------------+----------------+

What is with Altervista...I've now got 63 different ua's!!

[Crazy_Fool]

perhaps they're fed up with us trying to block scooter ? but then if they really want to spider our sites, why don't they put our sites in the SERPS ? i mean, not much point in them spidering otherwise ...

[theswampfox]

I've found my error log growing at an alarming rate. the 'scooter', or bot, robot, spider, whatever that's going through my site is requesting a bunch of stuff which is not even on my site.

trek12.sv.av.com - - [21/Nov/2001:16:12:22 -0500] "GET /sdb/en/html/kernel_hisax.html HTTP/1.0" 200 6283
trek12.sv.av.com - - [21/Nov/2001:16:12:27 -0500] "GET /sdb/en/html/plp.html HTTP/1.0" 200 4651
trek12.sv.av.com - - [21/Nov/2001:16:12:27 -0500] "GET /sdb/en/html/keylist.ATX.html HTTP/1.0" 200 2101
trek12.sv.av.com - - [21/Nov/2001:16:12:31 -0500] "GET /doc/support-db/sdb_e/ke_autofs.html HTTP/1.0" 200 4882
trek16.sv.av.com - - [21/Nov/2001:16:12:36 -0500] "GET /doc/sdb/de/html/keylist.LOCALHOST.html HTTP/1.0" 404 303
trek16.sv.av.com - - [21/Nov/2001:16:13:01 -0500] "GET /doc/sdb/en/html/tkman.html HTTP/1.0" 200 3549
trek12.sv.av.com - - [21/Nov/2001:16:13:14 -0500] "GET /sdb/es/html/swgkern_soundlive.html HTTP/1.0" 404 299
trek16.sv.av.com - - [21/Nov/2001:16:13:17 -0500] "GET /doc/de/html/maddin_runx.html HTTP/1.0" 404 293
trek12.sv.av.com - - [21/Nov/2001:16:13:21 -0500] "GET /sdb/en/html/keylist.GIGASTORAGE.html HTTP/1.0" 200 2146
trek16.sv.av.com - - [21/Nov/2001:16:13:47 -0500] "GET /doc/sdb/de/html/maddin_xserverrc.html HTTP/1.0" 404 302
trek16.sv.av.com - - [21/Nov/2001:16:13:48 -0500] "GET /doc/de/html/keylist.LIBVGA.html HTTP/1.0" 404 296
trek12.sv.av.com - - [21/Nov/2001:16:13:52 -0500] "GET /sdb/de/html/keylist.THINKPAD.html HTTP/1.0" 404 298
trek12.sv.av.com - - [21/Nov/2001:16:13:54 -0500] "GET /doc/support-db/sdb_e/ke_samba-60.html HTTP/1.0" 200 3362
trek16.sv.av.com - - [21/Nov/2001:16:13:55 -0500] "GET /doc/sdb/en/html/[e-mail] HTTP/1.0" 200 2354
trek12.sv.av.com - - [21/Nov/2001:16:14:01 -0500] "GET /doc/support-db/sdb_e/jd_pci_unkown.html HTTP/1.0" 200 5241
trek16.sv.av.com - - [21/Nov/2001:16:14:01 -0500] "GET ...

[littleman]

Welcome to wmw theswampfox.
Looks like it is looking at your SuSE documentation. You must have it publicly accessible.

[theswampfox]

littleman:

Yes, I run a SuSE OS, and the documentation is outside the root specified by the httpd. My robots.txt is:

# exclude help system from robots
User-agent: *
Disallow: /hilfe/ /manual/ /support-db/ /gif/
# but allow htdig to index our doc-tree
User-agent: susedig
Disallow:

This is the default for the version 7.0 I am running. With my little 768k sdsl, I've been hit pretty hard by the two av bots listed in the previous post. They pulled up every document on my server twice in three days, over 15,000 hits from 11/18 @ 01:00 to 11/21 @ 14:00. My problem is the mass of files requested which are not on my server.

Google comes through once a month like clockwork, goes through the pages orderly and pull only what is available through the ls command. This little grabber takes every page and is still not satisfied.

Don't get me wrong, my little site is non-commercial and I don't have much traffic at all, in fact it's a pretty dead site (with cause, not much worthwhile on it). Most of my hits are from worms like nimda and code red. I use it mostly to tag articles to see who is interested in my writings.

You should drop by <URL is in profile>. Drop by some time and check it out. Hey, I'd know it was you cause if you browse any, it would be the only hit of the week.

theswampfox

(edited by: Marcia at 5:19 am (gmt) on Nov. 24, 2001)

[Son_House]

so ... with all these new scooters, how do we block scooter ? do we just use "Scooter" in the robots.txt file or do we need to use the full UA for every scooter agent we see ?

That's a good question. If av is going to be nice, just Scooter should do the job. If not, robots.txt files are going to get huge. Anyone try blocking Scooter yet? If so did it work?

theswampfox, check your "stickymail". Link is at the top of the page.

[theswampfox]

might try blocking something like *av* and *scoot* (I believe you are allowed wildcards)