Welcome to WebmasterWorld Guest from 54.166.54.215

Forum Moderators: Ocean10000 & incrediBILL

Scooter/2.0 G.R.A.B. V1.1.0

   
2:22 pm on Jan 18, 2000 (gmt 0)

10+ Year Member



Hi Guys, I have been heavily crawled by this A.V agent. Any info would be great.

Thanks,
Pete

4:21 am on Jan 19, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



What is it specifically you need to know? Scooter is Altavista's standard spider. Getting crawled by them is a major accomplishment these days (congrats). Or is there some problem with the spidering?
10:43 am on Jan 19, 2000 (gmt 0)

10+ Year Member



Hi Brett,

Apologies for any ambiguity. I have been crawled by a number of A.V's spiders in the past. This is the first time that I have been crawled by the above agent (2.0 G.R.A.B. V1.1.0). As yet , none of the pages have been included. What would have sparked the spider to crawl this site. The site is new and I havent submitted any pages to A.V yet. As to my original question, what is this spiders purpose: crawl & include pages in the A.V database / record the number of links for link popularity purposes / collect images for A.V's image database e.t.c.

I hope this is clearer.
Thanks,
pete

12:29 am on Jan 20, 2000 (gmt 0)

10+ Year Member



That one has been over my site like gangbusters since AV unstuck. A little wierd though, kinda like a little kid just let out to play! Scooter looks to be very excited, looking for pages like index.htmindex2.htm (yup two pages run together)
3:55 pm on Feb 1, 2000 (gmt 0)

10+ Year Member



I have been hit by scooter several times in the past few weeks but each time they have only requested my robots.txt (which I don't have) and then took off. do you guys have any clue about what is going on? it did not bother to look at any of my other pages.
hb
5:04 pm on Feb 1, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Are you looking at raw log files? If so, look for HEAD requests by scooter. AV checks the document age by just doing HEAD checks on a document. If the doc hasn't updated since the last time it was there, it moves on.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month