htaccess - trying to block lwp-trivial returns my 404 error page

Trying to block lwp-trivial returns my 404 error page

[StaceyJ]

I am trying to block the LWP bot/worm or whatever it is called seen in my logs as lwp-trivial/1.41.

I have a functioning htaccess file but when I add

RewriteCond %{HTTP_USER_AGENT} ^LWP [NC]
RewriteRule .* - [F]

my index page and other pages not using a common template work fine, but when I try to go to a page that is generated from a template file I get either a 404 error or a script generated error about not being able to use the template specified.

I have a custom 404 page if that matters.

Do I also need to add something like

# Skip all following rules for robots.txt and 403 error page requests
RewriteRule (robot\.txt¦403-error-page\.html)$ - [L]

as I found in a post by jdMorgan approx. 4 months ago?

Any help would be greatly appreciated.

[jdMorgan]

Possibly -- Try it and see...

You should try to 'chase down' the 403 Error handling -- both your default handling and the handling (if any) within your 'template system'. The ErrorDocument 403 directive, if present, must point to a local filepath that exists (a local filepath, not not a URL), and I strongly recommend that 403, 500, and other critical errors *not* be handled by any scripting -- Otherwise, any error in the script or a bug in the script interpreter could result in an un-debuggable cascade of errors. Keep critical-error handling dirt-simple.

Jim

[StaceyJ]

Thanks. But since I don't do anything besides what the server does to handle 403's, I don't see where this would help (but that's why I'm asking :)

RewriteRule (robot\.txt¦403-error-page\.html)$ - [L]

Would I instead need to handle my custom 404 page with something like

RewriteRule (robots\.txt¦404-error-page\.html)$ - [L]

assuming my 404 page is named 404-error-page\.html?

Also, shouldn't it be robots\.txt with an "s"? Your previous post from a different topic didn't have the plural form.

edit - also, the script that handles the templates doesn't address 403's either.

[edited by: StaceyJ at 3:41 pm (utc) on July 15, 2008]