Welcome to WebmasterWorld Guest from 54.158.119.60

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Scooter/2.0 G.R.A.B. V1.1.0

     
2:22 pm on Jan 18, 2000 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 21, 1999
posts:370
votes: 0


Hi Guys, I have been heavily crawled by this A.V agent. Any info would be great.

Thanks,
Pete

4:21 am on Jan 19, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38066
votes: 15


What is it specifically you need to know? Scooter is Altavista's standard spider. Getting crawled by them is a major accomplishment these days (congrats). Or is there some problem with the spidering?
10:43 am on Jan 19, 2000 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 21, 1999
posts:370
votes: 0


Hi Brett,

Apologies for any ambiguity. I have been crawled by a number of A.V's spiders in the past. This is the first time that I have been crawled by the above agent (2.0 G.R.A.B. V1.1.0). As yet , none of the pages have been included. What would have sparked the spider to crawl this site. The site is new and I havent submitted any pages to A.V yet. As to my original question, what is this spiders purpose: crawl & include pages in the A.V database / record the number of links for link popularity purposes / collect images for A.V's image database e.t.c.

I hope this is clearer.
Thanks,
pete

RealSpices

12:29 am on Jan 20, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


That one has been over my site like gangbusters since AV unstuck. A little wierd though, kinda like a little kid just let out to play! Scooter looks to be very excited, looking for pages like index.htmindex2.htm (yup two pages run together)

hbhb

3:55 pm on Feb 1, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


I have been hit by scooter several times in the past few weeks but each time they have only requested my robots.txt (which I don't have) and then took off. do you guys have any clue about what is going on? it did not bother to look at any of my other pages.
hb
5:04 pm on Feb 1, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38066
votes: 15


Are you looking at raw log files? If so, look for HEAD requests by scooter. AV checks the document age by just doing HEAD checks on a document. If the doc hasn't updated since the last time it was there, it moves on.