Forum Moderators: open

Message Too Old, No Replies

vscooter

         

wilderness

2:07 pm on Oct 14, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



AltaVista has gone wacko and is violating robots.txt with the following UA.

"Scooter/3.3.vscooter"

Audun

7:35 pm on Oct 14, 2003 (gmt 0)

10+ Year Member



Can you confirm that this crawler is coming from Altavista?

wilderness

11:36 am on Oct 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Audun
Welcome to WebmasterWorld.

I almost didn't reply to your inquiry.
A search through the threads will provide my screename more than a few times. ;)
My profile says that I've been devoted to primarily this forum at WebmasterWorld for nearly two years. Although the other forums are quite interesting, my plate is rather full and I feel the need to limit myself to this forum, which proves beneficial to myself and my websites.

The IP was 216.39.50.64
Now I suppose it's possible that is a fake IP however AltaVista has some record in traveling beyond the suggestions in robots.txt. It has to do with their interpetation of sub folders and also the lack of control in their bot.

I've denied vscooter.

Don

jdMorgan

1:29 pm on Oct 15, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



vscooter has been messed up for quite a long time. IIRC, there is a thread from last year discussing its non-compliance w/robots.txt.

This is AV's "image" robot, so sites that see no benefit in getting their images indexed can safely block vscooter.

It's too bad that many companies don't realize that their robots are often their first contact with webmasters. The number of improperly-indentified (no contact/info in user-agent string), misbehaving, and broken robots is rather appalling.

Jim

Audun

3:58 pm on Oct 16, 2003 (gmt 0)

10+ Year Member



Wilderness,

no offence when i'm doubting your first statement, but there are a couple of spiders using fake UA. :)

I'm trying to build a complete (i know it impossible) database of web crawlers, so i need to ensure that the data i'm adding to it is correct.