Welcome to WebmasterWorld Guest from 23.22.220.37

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

New Altavista Behaviour...

libwww-perl/5.46 for av?

     
11:36 pm on May 30, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 19, 2000
posts:193
votes: 0


Hello, some altavista came by, and crawled my whole site, well-behaved, about 1 hit per second.

The only thing that struck me is the user-agent, libwww-perl. I would have thought AV would use something more ingeniuos that the perl-www library.
--
www.altavista.com - - [30/May/2001:22:20:54 +0200] "GET / HTTP/1.0" 200 6168 "-" "libwww-perl/5.46"

11:39 pm on May 30, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 19, 2000
posts:193
votes: 0


oh, something else: did NOT get the robots.txt file.
11:57 pm on May 30, 2001 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


What was the ip address?
7:44 am on May 31, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 26, 2000
posts:2176
votes: 0


Sounds like someone spoofing an AV useragent.
11:51 pm on June 4, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 19, 2000
posts:193
votes: 0


The line below the dashes is my actual logfile entry, its all I have (plus a few others with different GET requests)

June 1 to 3, I saw a few scooters coming by,
scooter3-scooter6.sv.av.com

they exhibited "normal" scooter-behaviour (ie. respected robots.txt, did not put more than one hit per second on my site, etc..)

Was just puzzled that av would use an (unadapted?) libwww-perl

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members