How can a webmaster block a user agent, like libwww-perl/5.805?

[Owdy]

I want to block that ( i heard it is bad :D )

I found this code:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} (^libwww-perl) [nc]
RewriteRule .* http://%{REMOTE_ADDR} [r=301,l]

What that actually do?

[wilderness]

There are numerous threads and explanations [google.com] in the archives.

[jdMorgan]

It redirects requests made by user-agents claiming to be libwww-perl back to their own IP address. It's rather a waste of time and internet bandwidth, because "bad guys" typically do not follow redirects.

It's often simpler and more efficient to simply deny the request and be done with it:

RewriteEngine on
#
RewriteCond %{HTTP_USER_AGENT} libwww-perl [NC]
RewriteRule ^ - [F]

Jim

[Owdy]

Thanks Jim! I try that. Is there any downsides blocking all?

[jdMorgan]

That would depend on whether you expect or allow any libwww-perl clients to access/download your site.

libwww-perl is simply a collection of HTTP-access routines written in PERL. It can be used to do anything that a browser or robot can do -- from checking your pages for updates, to making screenshots, to downloading your entire site. Whether this is good or bad depends on who is doing it and why, and on what you expect and allow on your site. So I cannot answer the that question with a simple yes or no.

I currently allow libwww-perl to access my site only from IP address ranges belonging to the search services of Yahoo (plus Yahoo properties AltaVista and Inktomi), Ask, IA Archiver, and Alexa. Because of the nature and location of your site, your needs may be very different.

Jim

[Owdy]

Thanks again! How do you know location of my site? ;D

[jdMorgan]

I don't. It was just a general statement, acknowledging the fact that sites in different marketing niches, regions, and countries can have very different needs.

Jim