| 2:04 pm on Nov 19, 2007 (gmt 0)|
LWP is a well-known perl library (and perl module and and lwp-request which is a simple command line user agent).
it could be just about anything - a homemade bot or custom browser, somebody running a script - it's a very generic tool.
| 11:00 pm on Nov 19, 2007 (gmt 0)|
Okay, but why would someone want to run a bot requesting those URLs from different hosts?
[edited by: Patrick_Taylor at 11:02 pm (utc) on Nov. 19, 2007]
| 1:43 am on Nov 20, 2007 (gmt 0)|
It could be a proxy server, which may or may not be malicious, but some are known to zap your pages in the SERPs if they aren't handled correctly - IIRC they can be considered duplicate content by Google and end up getting the better ranking. There was a whole thread here concerning this problem, if this is the case for you its worth a read.
| 1:44 am on Nov 20, 2007 (gmt 0)|
does that look at all like a url you might serve?
| 11:29 am on Nov 20, 2007 (gmt 0)|
There's this thread -> [webmasterworld.com...]
Quite helpful. I suppose the safest thing is to serve a 403 to a bot that requests an URL like:
And they're doing it dozens of times a day.
[edited by: Patrick_Taylor at 11:31 am (utc) on Nov. 20, 2007]