New AV spider - Scooter-3.0.g0-test

[mark_roach]

Got this one in my logs yesterday:

209.73.164.127 - - [26/Mar/2001:22:08:42 -0500] "GET /robots.txt HTTP/1.0" 200 616 "-" "Scooter-3.0.g0-test"

[Marcia]

Just a little while ago:

209.73.164.132 - - [27/Mar/2001:06:03:36 -0500] "GET /robots.txt HTTP/1.0" 404 216 "-" "Scooter-3.0.d0"
209.73.164.132 - - [27/Mar/2001:06:03:36 -0500] "GET / HTTP/1.0" 200 11100 "-" "Scooter-3.0.d0"

[ulstrup]

Got it too

2001-03-27 02:13:30 209.73.164.132 Scooter-3.0.d0 -

2001-03-27 02:13:33 209.73.164.132 Scooter-3.0.d0 -

2001-03-27 04:12:09 209.73.164.132 Scooter-3.0.d0 -

19 pages all edited within the last month. All "level one" pages (links on index)

2001-03-27 06:26:33 209.73.164.132 Scooter-3.0.d0 -

2001-03-27 18:20:33 209.73.164.132 Scooter-3.0.d0 -

11 pages all edited within the last month, some dublicate crawls, also "level 2 and 3"

2001-03-27 18:24:19 209.73.164.132 Scooter-3.0.d0 -

21 pages, mostly non previously spidered pages, all mostly not edited for some time, level 2 and 3.

The site is about 3 years old and has good linkpop, most pages in the first "big" crawl (the newly edited pages) have been submitted to AV.com, AV.europe and AV.asia within the last month.

Well I don't complain :)

First big hug from Scooter in a looong time :))

I'll let you know when and where the pages are indexed, my guess is that Scooter 3 is European - well, a few of them actually are indexed in edited versions in av.com.

But the big news (for me atleast) is that Scooter crawls again ! Hope he hits your place too :)

Message has been updated.

Edited by: ulstrup

[littleman]

I've also been hit by it lightly.

[ulstrup]

Deep Crawl yes it is :)

2001-03-28 03:46:15 209.73.164.126 Scooter-3.0.g0-11 -
2001-03-28 03:50:19 209.73.164.127 Scooter-3.0.g12-23 -
2001-03-28 04:15:46 209.73.164.129 Scooter-3.0.g24-35 -

These 3 Scooters hit randomly 23 pages in total...

[Brett_Tabke]

The question is if this is a data mining crawl or the real deal.

[PeteU]

my feeling is it maybe the real thing, data mining is usually more extensive and goes into all domains, this crawl only reqests pages from domains not previously banned by AV

[jk3210]

Whoa!

I'd given up on AV and paid no attention to it recently, but after reading this thread I ran a check and scooter has been walking all over my site all month!

3/1 to 3/17 scooter crawled ALL the out-of-date pages I've been trying to get removed for quite a while.

3/21 to present scooter has crawled about 250 pages.

I guess when you act like you really don't care, it just drives these spiders wild!