homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

New AV spider - Scooter-3.0.g0-test

 12:18 pm on Mar 27, 2001 (gmt 0)

Got this one in my logs yesterday: - - [26/Mar/2001:22:08:42 -0500] "GET /robots.txt HTTP/1.0" 200 616 "-" "Scooter-3.0.g0-test"



 12:38 pm on Mar 27, 2001 (gmt 0)

Just a little while ago: - - [27/Mar/2001:06:03:36 -0500] "GET /robots.txt HTTP/1.0" 404 216 "-" "Scooter-3.0.d0" - - [27/Mar/2001:06:03:36 -0500] "GET / HTTP/1.0" 200 11100 "-" "Scooter-3.0.d0"


 3:29 pm on Mar 27, 2001 (gmt 0)

Got it too

2001-03-27 02:13:30 Scooter-3.0.d0 -

2001-03-27 02:13:33 Scooter-3.0.d0 -

2001-03-27 04:12:09 Scooter-3.0.d0 -

19 pages all edited within the last month. All "level one" pages (links on index)

2001-03-27 06:26:33 Scooter-3.0.d0 -

2001-03-27 18:20:33 Scooter-3.0.d0 -

11 pages all edited within the last month, some dublicate crawls, also "level 2 and 3"

2001-03-27 18:24:19 Scooter-3.0.d0 -

21 pages, mostly non previously spidered pages, all mostly not edited for some time, level 2 and 3.

The site is about 3 years old and has good linkpop, most pages in the first "big" crawl (the newly edited pages) have been submitted to AV.com, AV.europe and AV.asia within the last month.

Well I don't complain :)

First big hug from Scooter in a looong time :))

I'll let you know when and where the pages are indexed, my guess is that Scooter 3 is European - well, a few of them actually are indexed in edited versions in av.com.

But the big news (for me atleast) is that Scooter crawls again ! Hope he hits your place too :)

Message has been updated.

Edited by: ulstrup


 7:25 pm on Mar 27, 2001 (gmt 0)

I've also been hit by it lightly.


 8:06 am on Mar 28, 2001 (gmt 0)

Deep Crawl yes it is :)

2001-03-28 03:46:15 Scooter-3.0.g0-11 -
2001-03-28 03:50:19 Scooter-3.0.g12-23 -
2001-03-28 04:15:46 Scooter-3.0.g24-35 -

These 3 Scooters hit randomly 23 pages in total...


 12:19 pm on Mar 28, 2001 (gmt 0)

The question is if this is a data mining crawl or the real deal.


 12:14 am on Mar 30, 2001 (gmt 0)

my feeling is it maybe the real thing, data mining is usually more extensive and goes into all domains, this crawl only reqests pages from domains not previously banned by AV


 9:56 pm on Mar 31, 2001 (gmt 0)


I'd given up on AV and paid no attention to it recently, but after reading this thread I ran a check and scooter has been walking all over my site all month!

3/1 to 3/17 scooter crawled ALL the out-of-date pages I've been trying to get removed for quite a while.

3/21 to present scooter has crawled about 250 pages.

I guess when you act like you really don't care, it just drives these spiders wild!

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved