homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Scooter/2.0 G.R.A.B. V1.1.0

 2:22 pm on Jan 18, 2000 (gmt 0)

Hi Guys, I have been heavily crawled by this A.V agent. Any info would be great.




 4:21 am on Jan 19, 2000 (gmt 0)

What is it specifically you need to know? Scooter is Altavista's standard spider. Getting crawled by them is a major accomplishment these days (congrats). Or is there some problem with the spidering?


 10:43 am on Jan 19, 2000 (gmt 0)

Hi Brett,

Apologies for any ambiguity. I have been crawled by a number of A.V's spiders in the past. This is the first time that I have been crawled by the above agent (2.0 G.R.A.B. V1.1.0). As yet , none of the pages have been included. What would have sparked the spider to crawl this site. The site is new and I havent submitted any pages to A.V yet. As to my original question, what is this spiders purpose: crawl & include pages in the A.V database / record the number of links for link popularity purposes / collect images for A.V's image database e.t.c.

I hope this is clearer.


 12:29 am on Jan 20, 2000 (gmt 0)

That one has been over my site like gangbusters since AV unstuck. A little wierd though, kinda like a little kid just let out to play! Scooter looks to be very excited, looking for pages like index.htmindex2.htm (yup two pages run together)


 3:55 pm on Feb 1, 2000 (gmt 0)

I have been hit by scooter several times in the past few weeks but each time they have only requested my robots.txt (which I don't have) and then took off. do you guys have any clue about what is going on? it did not bother to look at any of my other pages.


 5:04 pm on Feb 1, 2000 (gmt 0)

Are you looking at raw log files? If so, look for HEAD requests by scooter. AV checks the document age by just doing HEAD checks on a document. If the doc hasn't updated since the last time it was there, it moves on.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved