Scooter/2.0 G.R.A.B. V1.1.0

[Pete]

Hi Guys, I have been heavily crawled by this A.V agent. Any info would be great.

Thanks,
Pete

[Brett_Tabke]

What is it specifically you need to know? Scooter is Altavista's standard spider. Getting crawled by them is a major accomplishment these days (congrats). Or is there some problem with the spidering?

[Pete]

Hi Brett,

Apologies for any ambiguity. I have been crawled by a number of A.V's spiders in the past. This is the first time that I have been crawled by the above agent (2.0 G.R.A.B. V1.1.0). As yet , none of the pages have been included. What would have sparked the spider to crawl this site. The site is new and I havent submitted any pages to A.V yet. As to my original question, what is this spiders purpose: crawl & include pages in the A.V database / record the number of links for link popularity purposes / collect images for A.V's image database e.t.c.

I hope this is clearer.
Thanks,
pete

[RealSpices]

That one has been over my site like gangbusters since AV unstuck. A little wierd though, kinda like a little kid just let out to play! Scooter looks to be very excited, looking for pages like index.htmindex2.htm (yup two pages run together)

[hbhb]

I have been hit by scooter several times in the past few weeks but each time they have only requested my robots.txt (which I don't have) and then took off. do you guys have any clue about what is going on? it did not bother to look at any of my other pages.
hb

[Brett_Tabke]

Are you looking at raw log files? If so, look for HEAD requests by scooter. AV checks the document age by just doing HEAD checks on a document. If the doc hasn't updated since the last time it was there, it moves on.