Scooter going crazy

at least it seems like that to me

[rankboy]

Hi everyone,
I have been seeing something very weird in my logs lately. I have over 1800 hits by Scooter across some of my domains! It's crawling index pages and beyond. I'm wondering if anyone else is seeing this going on. I haven't used any PFI on any of these domains either and I also never submitted them to av (it's such a pain now with those darn ransom notes) Maybe av is going to mounting a comeback pretty soon?

[lazerzubb]

Is it crawling dynamic pages?
Because then it might be trapped in a "Spider trapp"

[rankboy]

nope, it's not stuck on any one domain or page, it's going through all the different domains and different pages on those domains. I'm using static pages anyway so I don't think it's getting tripped up. Hopefully all these pages show up in the next index.

[bobriggs]

I've noticed Scooter being very busy across all of my domains for the last couple of days, also. And Slurp seems to be branching out into new territory.

My first thought was that these 2 might be feeling some heat from google.

[Axacta]

Yesterday Scooter came and spidered all of my pages. This is good because I have been waiting quite a while to be listed in AV, but today I see Scooter is respidering the whole site again. What could this mean? Is it just checking to see if anthing has changed so it can decide on a respidering schedule?

[zeb]

Scooter is right now working hard on one of my sites that has never been spidered before.
Does anybody know how long it takes of AV to list pages after the crawl?

The traffic from AV to an older site of mine has doubled since March and is now about 1/3 of Google's traffic.

This is very promising!

[groul]

My site was spidered, too though I never paid them anything, and haven't even used the Free Submission tool.

[Everyman]

We've finally had it with AltaVista. It's the only bot I've seen that is consistently abusive. A GET twice per second, and then it might start all over, just for the hell of it. It can get caught in furious loops on static files. We have over 120,000 pages on all the (independent) domains on our Class C. By the way, everyone on our Class C is nonprofit.

We decided that the referrals are not worth it. We now throw everything from 64.152.75.* into the bit bucket at our router.

We were watching when AV first crawled our server in August or September of 1995 ("What's this? Some sort of automatic thing?") Very exciting to see the first bot come through.

But it's been downhill now for the last few years. Even by summer of 1999, we were occasionally using Disallow: / on scooter, because who needs 'em?

Good riddance. Out of sight, out of mind. End of AV for us.

[mbauser2]

I have exactly the opposite problem: Scooter stops by my site every couple of days, grabs "robots.txt", and leaves. It never requests anything else.

(After 7 years of being listed for free, my pages did a strange, slow fadeout in the AV search results. The homepage disappeared first, leaving the deeper pages in the index. They finally all my pages dropped out a couple of months ago.)

[przero2]

I checked that scooter and mercator (is this also AV bot) passed by our site recently. When is an update expected?. Also, when did AV update last??. Thanks.

[Marcia]

>mercator (is this also AV bot)

Yes, przero2, Mercator is AV.

mbauser2, have you run your site through the robots.txt validator [searchengineworld.com]? Also, I can't remember whether it was AV or Ink, but I once read about a problem someone had with one of them crawling their sites because of the robots meta tag syntax. They had content="ALL" and had to change it to content="index,follow"

[Key_Master]

przero2,

You might want to WHOIS that IP. The Mercator agent I've been seeing lately hasn't been coming from AV.

[Olaf]

I'm not sure Im going to take your approach Everyman :-) Considering that AV still has around 6% market share Im willing to see what that is in referals. I admit that scooter is pretty ill behaved (1 page per sec on average). For the past 5 days Scooter has been going wild on my site. He has grabbed 92.000 pages to date and I have no idea when he will have his fill. Since he has minimal effects on the site or database I´m willing to let him finish.

I am curious about the % refs I will get from AV after they update their indices.

And to top it all, I have never paid them or anyone else a dime.

For years now I have only had the root /index.html in their index. Refs have been 3-4 a week. Since I do not overload keywords on that page I usually do not get a targeted audience from AV.

[buffy]

At the risk of sounding silly, where are you guys finding this info on spiders? From logs, and if it is logs, do you spend all day going through them. Pony up and tell us the method.

[rankboy]

Buffy,
I don't know about everyone else but i move all my logs over to a mysql database and then run quick queries to find out what's going on. The database allows you to search for things like: show me every hit that has googlebot or slurp as a user agent.