| 4:21 am on Jan 19, 2000 (gmt 0)|
What is it specifically you need to know? Scooter is Altavista's standard spider. Getting crawled by them is a major accomplishment these days (congrats). Or is there some problem with the spidering?
| 10:43 am on Jan 19, 2000 (gmt 0)|
Apologies for any ambiguity. I have been crawled by a number of A.V's spiders in the past. This is the first time that I have been crawled by the above agent (2.0 G.R.A.B. V1.1.0). As yet , none of the pages have been included. What would have sparked the spider to crawl this site. The site is new and I havent submitted any pages to A.V yet. As to my original question, what is this spiders purpose: crawl & include pages in the A.V database / record the number of links for link popularity purposes / collect images for A.V's image database e.t.c.
I hope this is clearer.
| 12:29 am on Jan 20, 2000 (gmt 0)|
That one has been over my site like gangbusters since AV unstuck. A little wierd though, kinda like a little kid just let out to play! Scooter looks to be very excited, looking for pages like index.htmindex2.htm (yup two pages run together)
| 3:55 pm on Feb 1, 2000 (gmt 0)|
I have been hit by scooter several times in the past few weeks but each time they have only requested my robots.txt (which I don't have) and then took off. do you guys have any clue about what is going on? it did not bother to look at any of my other pages.
| 5:04 pm on Feb 1, 2000 (gmt 0)|
Are you looking at raw log files? If so, look for HEAD requests by scooter. AV checks the document age by just doing HEAD checks on a document. If the doc hasn't updated since the last time it was there, it moves on.