homepage Welcome to WebmasterWorld Guest from 54.166.14.218
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
New AV spider - Scooter-3.0.g0-test
mark_roach

10+ Year Member



 
Msg#: 487 posted 12:18 pm on Mar 27, 2001 (gmt 0)

Got this one in my logs yesterday:

209.73.164.127 - - [26/Mar/2001:22:08:42 -0500] "GET /robots.txt HTTP/1.0" 200 616 "-" "Scooter-3.0.g0-test"

 

Marcia

WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 487 posted 12:38 pm on Mar 27, 2001 (gmt 0)

Just a little while ago:

209.73.164.132 - - [27/Mar/2001:06:03:36 -0500] "GET /robots.txt HTTP/1.0" 404 216 "-" "Scooter-3.0.d0"
209.73.164.132 - - [27/Mar/2001:06:03:36 -0500] "GET / HTTP/1.0" 200 11100 "-" "Scooter-3.0.d0"

ulstrup

10+ Year Member



 
Msg#: 487 posted 3:29 pm on Mar 27, 2001 (gmt 0)

Got it too

2001-03-27 02:13:30 209.73.164.132 Scooter-3.0.d0 -

2001-03-27 02:13:33 209.73.164.132 Scooter-3.0.d0 -

2001-03-27 04:12:09 209.73.164.132 Scooter-3.0.d0 -

19 pages all edited within the last month. All "level one" pages (links on index)

2001-03-27 06:26:33 209.73.164.132 Scooter-3.0.d0 -

2001-03-27 18:20:33 209.73.164.132 Scooter-3.0.d0 -

11 pages all edited within the last month, some dublicate crawls, also "level 2 and 3"

2001-03-27 18:24:19 209.73.164.132 Scooter-3.0.d0 -

21 pages, mostly non previously spidered pages, all mostly not edited for some time, level 2 and 3.

The site is about 3 years old and has good linkpop, most pages in the first "big" crawl (the newly edited pages) have been submitted to AV.com, AV.europe and AV.asia within the last month.

Well I don't complain :)

First big hug from Scooter in a looong time :))

I'll let you know when and where the pages are indexed, my guess is that Scooter 3 is European - well, a few of them actually are indexed in edited versions in av.com.

But the big news (for me atleast) is that Scooter crawls again ! Hope he hits your place too :)

Message has been updated.

Edited by: ulstrup

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 487 posted 7:25 pm on Mar 27, 2001 (gmt 0)

I've also been hit by it lightly.

ulstrup

10+ Year Member



 
Msg#: 487 posted 8:06 am on Mar 28, 2001 (gmt 0)

Deep Crawl yes it is :)

2001-03-28 03:46:15 209.73.164.126 Scooter-3.0.g0-11 -
2001-03-28 03:50:19 209.73.164.127 Scooter-3.0.g12-23 -
2001-03-28 04:15:46 209.73.164.129 Scooter-3.0.g24-35 -

These 3 Scooters hit randomly 23 pages in total...

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 487 posted 12:19 pm on Mar 28, 2001 (gmt 0)

The question is if this is a data mining crawl or the real deal.

PeteU

10+ Year Member



 
Msg#: 487 posted 12:14 am on Mar 30, 2001 (gmt 0)

my feeling is it maybe the real thing, data mining is usually more extensive and goes into all domains, this crawl only reqests pages from domains not previously banned by AV

jk3210

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 487 posted 9:56 pm on Mar 31, 2001 (gmt 0)

Whoa!

I'd given up on AV and paid no attention to it recently, but after reading this thread I ran a check and scooter has been walking all over my site all month!

3/1 to 3/17 scooter crawled ALL the out-of-date pages I've been trying to get removed for quite a while.

3/21 to present scooter has crawled about 250 pages.

I guess when you act like you really don't care, it just drives these spiders wild!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved