homepage Welcome to WebmasterWorld Guest from 54.161.236.229
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
New AV spider - Scooter-3.0.g0-test
mark_roach




msg:399704
 12:18 pm on Mar 27, 2001 (gmt 0)

Got this one in my logs yesterday:

209.73.164.127 - - [26/Mar/2001:22:08:42 -0500] "GET /robots.txt HTTP/1.0" 200 616 "-" "Scooter-3.0.g0-test"

 

Marcia




msg:399705
 12:38 pm on Mar 27, 2001 (gmt 0)

Just a little while ago:

209.73.164.132 - - [27/Mar/2001:06:03:36 -0500] "GET /robots.txt HTTP/1.0" 404 216 "-" "Scooter-3.0.d0"
209.73.164.132 - - [27/Mar/2001:06:03:36 -0500] "GET / HTTP/1.0" 200 11100 "-" "Scooter-3.0.d0"

ulstrup




msg:399706
 3:29 pm on Mar 27, 2001 (gmt 0)

Got it too

2001-03-27 02:13:30 209.73.164.132 Scooter-3.0.d0 -

2001-03-27 02:13:33 209.73.164.132 Scooter-3.0.d0 -

2001-03-27 04:12:09 209.73.164.132 Scooter-3.0.d0 -

19 pages all edited within the last month. All "level one" pages (links on index)

2001-03-27 06:26:33 209.73.164.132 Scooter-3.0.d0 -

2001-03-27 18:20:33 209.73.164.132 Scooter-3.0.d0 -

11 pages all edited within the last month, some dublicate crawls, also "level 2 and 3"

2001-03-27 18:24:19 209.73.164.132 Scooter-3.0.d0 -

21 pages, mostly non previously spidered pages, all mostly not edited for some time, level 2 and 3.

The site is about 3 years old and has good linkpop, most pages in the first "big" crawl (the newly edited pages) have been submitted to AV.com, AV.europe and AV.asia within the last month.

Well I don't complain :)

First big hug from Scooter in a looong time :))

I'll let you know when and where the pages are indexed, my guess is that Scooter 3 is European - well, a few of them actually are indexed in edited versions in av.com.

But the big news (for me atleast) is that Scooter crawls again ! Hope he hits your place too :)

Message has been updated.

Edited by: ulstrup

littleman




msg:399707
 7:25 pm on Mar 27, 2001 (gmt 0)

I've also been hit by it lightly.

ulstrup




msg:399708
 8:06 am on Mar 28, 2001 (gmt 0)

Deep Crawl yes it is :)

2001-03-28 03:46:15 209.73.164.126 Scooter-3.0.g0-11 -
2001-03-28 03:50:19 209.73.164.127 Scooter-3.0.g12-23 -
2001-03-28 04:15:46 209.73.164.129 Scooter-3.0.g24-35 -

These 3 Scooters hit randomly 23 pages in total...

Brett_Tabke




msg:399709
 12:19 pm on Mar 28, 2001 (gmt 0)

The question is if this is a data mining crawl or the real deal.

PeteU




msg:399710
 12:14 am on Mar 30, 2001 (gmt 0)

my feeling is it maybe the real thing, data mining is usually more extensive and goes into all domains, this crawl only reqests pages from domains not previously banned by AV

jk3210




msg:399711
 9:56 pm on Mar 31, 2001 (gmt 0)

Whoa!

I'd given up on AV and paid no attention to it recently, but after reading this thread I ran a check and scooter has been walking all over my site all month!

3/1 to 3/17 scooter crawled ALL the out-of-date pages I've been trying to get removed for quite a while.

3/21 to present scooter has crawled about 250 pages.

I guess when you act like you really don't care, it just drives these spiders wild!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved