Welcome to WebmasterWorld Guest from 54.242.224.250

Forum Moderators: DixonJones & mademetop

Message Too Old, No Replies

Frequent visits from user agent libwww-perl/5.805

What's this?

     
4:59 pm on Nov 18, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 30, 2003
posts: 932
votes: 0


Two pages on one of my websites have begun to receive frequent visits from bots with different host names but the user agent is always like:

libwww-perl/5.805
libwww-perl/5.64
libwww-perl/5.63

The requested URLs are typically:

ht*p://www.mysite.com/examplepage/index.php?page=ht*p://www.somesite.com/home/images/can?
ht*p://www.mysite.com//index.php?page=ht*p://somesite.nl/id.txt?

(my asterisks)

I've now redirected all such requests to remove index.php and the query string, but I'd be interested to know what's going on with these visits. Is it some sort of mischief?

It seems like a tracking system (or something).

2:04 pm on Nov 19, 2007 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10563
votes: 15


LWP is a well-known perl library (and perl module and and lwp-request which is a simple command line user agent).
it could be just about anything - a homemade bot or custom browser, somebody running a script - it's a very generic tool.
11:00 pm on Nov 19, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 30, 2003
posts: 932
votes: 0


Okay, but why would someone want to run a bot requesting those URLs from different hosts?

[edited by: Patrick_Taylor at 11:02 pm (utc) on Nov. 19, 2007]

1:43 am on Nov 20, 2007 (gmt 0)

Junior Member

10+ Year Member

joined:May 6, 2005
posts:134
votes: 0


It could be a proxy server, which may or may not be malicious, but some are known to zap your pages in the SERPs if they aren't handled correctly - IIRC they can be considered duplicate content by Google and end up getting the better ranking. There was a whole thread here concerning this problem, if this is the case for you its worth a read.
1:44 am on Nov 20, 2007 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10563
votes: 15


does that look at all like a url you might serve?
11:29 am on Nov 20, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 30, 2003
posts: 932
votes: 0


There's this thread -> [webmasterworld.com...]

Quite helpful. I suppose the safest thing is to serve a 403 to a bot that requests an URL like:

ht*p://www.mysite.com//index.php?page=ht*p://somesite.nl/id.txt?

And they're doing it dozens of times a day.

[edited by: Patrick_Taylor at 11:31 am (utc) on Nov. 20, 2007]

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members