Scooter out in full server abuse mode...

[Brett_Tabke]

Anyone else getting their servers beat up by scooter? Thousands upon thousands of hits to some sites. Wide range of sites so I imagine it has been happening to everyone.

[makemetop]

1909 in past 24 hours - this has been the heaviest assault yet by Scooter. Though it has been hitting sites pretty heavily and consistently since Brett's first post in this thread.

[mark_roach]

Twice as bad as before. 30000 pages yesterday. I didn't know I had that many !

[caine]

AV is going mad.

Trek bots are crawling like mad, with more gets yesterday, than scooter(s) and treks(s) last month.

[zero6]

Has anybody had any pages listed yet? This is the second time I've had scooter go through my site but nothing on AV yet.

[conor]

I have one site that AV normaally hated but noe it seems to be loving it.... even dynamic pages !!!!

[FreeBee]

Several site logs show a bit hit on 9 and 10, as deep as the crawls from 3rd week July.

The only pages we've found actually added to the database for a new site were from a small batch submitted before the PFI date and some up to 20 and 21 June but none of the more recent submissions. Also, not one of several hundred new pages repeatedly spidered by Scooter is getting in - it looks like you have to manually submit (or pay!)

[Macguru]

Average for 223 sites. Treck crawling like crazy, pagecount doubled, results dropped.

[mburgess]

I'm a little new to the whole spider and hits thing. So, all of this that I have been reading, this is a bad thing for the spider to be coming through and hitting pages like this. Could this be related to a drop in rankings? On August 1, and in previous months, I had 13 key phrases listed on page one. Today, August 13, I only have 4. What's going on and how can I change this?

[banav]

hi,
any ideas on how can i limit the frequency with which scooter crawls my site? brett, were you able to successfully ban them?
thanks

[Great1]

80k pages spidered in 11 and a half hours. It's bloody stupid, no one in the office can get out, no visitors can get in.

[Brett_Tabke]

I banned them temporarily via an htaccess file exclusion. They were hitting heavy at peak hours and no one could use the site.

Just for the record, AV techs are aware of the problem and have been taking steps to reduce loads.

[jquerin]

Scooter has hit our site between 6-5-01 to 8-28-01 with a total of 1931 hits. Then on Monday 8-27-01 we got hit a total of 1477 hits in about 5 hours and 18 mins, which ends up being about 5 a minute. I haven't seen any hits from Trek. Except a Lotus Web trek hit. So far today 8-29-01 there is no hits from this scooter bot... anybody else seeing any activity or has AV taken it offline?

[msgraph]

>>>I banned them temporarily via an htaccess file exclusion. They were hitting heavy at peak hours and no one could use the site.

Any word on a turnaround time for them to fix this problem Brett?

They've locked up one server today grabbing 300k of information per second. Seriously thinking of disallowing everything except the root index pages.

[Crazy_Fool]

Thought I'd better post here as well ... had a new scooter UA on one of my sites today and it ripped through 25 pages in 17 seconds.
Those of you having problems with scooter might want to block it.

209.73.162.151
Scooter_trk15-3.0.3

[caine]

spotted trek7 yesterday, doing the rounds though not as prolific as july/augusts summer madness

[jeremy goodrich]

Since June, they have been spidering my site frequently, and at times, quite a load (even for a small site).

So as of last week, I started banning them in my .htaccess file...they now get a 403 every time they hit my site (and they have 3 times since). I don't know about the rest of the community, but I expect something in exchange for the frustration of watching the thing slow down my site, and fill my logs with useless requests.

If they index me (and they should have at least 10 copies of the whole bloody site) I'll consider taking off the ban. Till then, I'm not letting altavista have my stuff. It wasn't time for their server, since they won't index anyway, and it waste's my time, because the only reason i go to Atavista [altavista.com] anymore is to see if the pages got listed. I'm not about to actually "search" for anything there.

[Crazy_Fool]

well, i can hardly believe it ... i finally got a site visitor from altavista ...

sort of ....

the visitor actually came from ask.co.uk which showed a result from altavista.com for an 8 word search term ....

but it's still an altavista result isn't it? please, someone tell me its an altavista result ?? i'll be ever so disappointed if it isnt ..... LOL

[msgraph]

The reply from the big AV.

"We don't consider this activity an "attack" or "security breach" unless any of the following occur: "

Well 1 and 5 is happening at this moment.

The accesses to your machine are most likely the result of Scooter's
normal operation. Scooter usually finds your site by following a link
from another page somewhere on the Internet. We don't consider this
activity an "attack" or "security breach" unless any of the following
occur:

1) Requests for the same URL over and over, or opening many connections
at once, denying service to other legitimate users. (Scooter opens only
one connection at a time to any given server and waits before visiting
again).

2) "Scanning" multiple ports in sequence to find all services. (Scooter
only visits URLs it has heard of before, and doesn't try to "guess"
URLs. Scooter will usually come to port 80 unless a link was found or
submitted with an alternate port number).

3) "Cracking" or trying to guess passwords. (Scooter will never provide
an http password, even if one is given in the URL, and will never try to
guess passwords).

4) Using any protocol besides HTTP. (Scooter may occasionally connect
to a port that is not a web server, if it finds something that looks
like a URL containing a colon followed by that number. If the service
at that port doesn't respond to HTTP commands, Scooter will break the
connection, and will not attempt to negotiate some other protocol).

5)Ignoring your robots.txt (Scooter requests "/robots.txt" first and
respects the directions given there, and scooter will periodically
request a new copy of /robots.txt in case it is new or has changed).

By far the best way to get Scooter to leave your site alone is to write
a robots.txt and place it at the top level of the web documents tree.
Scooter (and numerous other robot crawlers) will recognize and obey
these directions. Scooter has no way of knowing whether you consider
something "private" or "unauthorized", it simply follows anything that
looks like a link to a web page. If your host is an "internal use only"
server, you may instead choose to block web access at the router using a
firewall or screen, or at your web server by restricting access by IP
number or subnet. Alternatively, you can establish a "public" service on
one port, and a "private" service on another port, using IP address
rules. This is a more reliable way to protect sensitive or confidential
documents, because it blocks out all access, not just robots access. If
Scooter cannot reach your site because it is blocked at the router, it
will eventually give up and stop trying.

[backus]

Scooter's going mental. It's all over me, gobbling up everything in site! It ate everything, including .css files I have blocked in the robots.txt!!!!