homepage Welcome to WebmasterWorld Guest from 54.211.230.186
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
New Altavista Behaviour...
libwww-perl/5.46 for av?
skirril

10+ Year Member



 
Msg#: 725 posted 11:36 pm on May 30, 2001 (gmt 0)

Hello, some altavista came by, and crawled my whole site, well-behaved, about 1 hit per second.

The only thing that struck me is the user-agent, libwww-perl. I would have thought AV would use something more ingeniuos that the perl-www library.
--
www.altavista.com - - [30/May/2001:22:20:54 +0200] "GET / HTTP/1.0" 200 6168 "-" "libwww-perl/5.46"

 

skirril

10+ Year Member



 
Msg#: 725 posted 11:39 pm on May 30, 2001 (gmt 0)

oh, something else: did NOT get the robots.txt file.

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 725 posted 11:57 pm on May 30, 2001 (gmt 0)

What was the ip address?

WebGuerrilla

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 725 posted 7:44 am on May 31, 2001 (gmt 0)

Sounds like someone spoofing an AV useragent.

skirril

10+ Year Member



 
Msg#: 725 posted 11:51 pm on Jun 4, 2001 (gmt 0)

The line below the dashes is my actual logfile entry, its all I have (plus a few others with different GET requests)

June 1 to 3, I saw a few scooters coming by,
scooter3-scooter6.sv.av.com

they exhibited "normal" scooter-behaviour (ie. respected robots.txt, did not put more than one hit per second on my site, etc..)

Was just puzzled that av would use an (unadapted?) libwww-perl

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved