.htaccess deny not working

[macavity]

Can anyone spot an obvious problem with the following .htaccess file? The "deny from" lines are not having the required effect, i.e. they're not blocking visitors from the addresses in question (I've removed the actual IP addresses but they're all in standard IPv4 dot-decimal notation):

# Set up so that Allows override Denys (required to enable error pages to function)
<FilesMatch ".*">
Order deny,allow
</FilesMatch>

# Ban IP adresses and address ranges here

Deny from #*$!.#*$!.#*$!.#*$!
Deny from #*$!.#*$!.#*$!.#*$!
Deny from #*$!.#*$!.#*$!.#*$!

# Deny many common filetypes except .php
<FilesMatch "\.(inc|tpl|h|ihtml|sql|ini|conf|bin|spd|theme|module)$">
Deny from all
</FilesMatch>

# disable access to config.php and .ht* from a browser
<FilesMatch "^(config\.php|\.ht)">
Deny from all
</FilesMatch>

<FilesMatch "error\.(php|gif)">
Allow from all
</FilesMatch>

## Block Bad Bots by User-Agent
SetEnvIfNoCase User-Agent "Havij" bad_bot

<Files *>
Deny from env=bad_bot
</Files>

[wilderness]

Don't believe I've ever seen an htaccess with so many files-containers.

You need to start from scratch and repair all that crap.

Deny from env=bad_bot

works with Order deny,allow

and should be in close unison.

don't know how you expect anybody to read this?
It's impossible to tell if you have something missing, or something added unnecessarily.
Deny from #*$!.#*$!.#*$!.#*$!
Deny from #*$!.#*$!.#*$!.#*$!
Deny from #*$!.#*$!.#*$!.#*$!

It's perfectly acceptable to use IP ranges through the CLASS C's and then ONLY obscure the Class D ranges.

[g1smd]

Correct me if I am wrong, but I believe that only the rule in the last container will apply.

[lucy24]

You mean the wild-card * subsumes all the earlier named or matched files, so you end up with only one "Deny from" rule-- which only applies to one user-agent?

Whoops! :)

The <Directory>, <Files>, and <Location> directives can each use shell-style wildcard characters as in fnmatch from the C standard library. The character "*" matches any sequence of characters, "?" matches any single character, and "[seq]" matches any character in seq. The "/" character will not be matched by any wildcard; it must be specified explicitly.

Says the horse's mouth [httpd.apache.org] under "Wildcards and Regular Expressions" within "Filesystem and Webspace".

You've got way too many FilesMatch directives in there. And the "required to enable error pages to function" reasoning doesn't sound right.

:: shuffling papers ::

I've got

<FilesMatch (forbidden|goaway)\.html>
Order Allow,Deny
Allow from all
</FilesMatch>

<Files robots.txt>
Order Allow,Deny
Allow from all
</Files>

followed by a bunch of SetEnvIf lines, followed by

Order Allow,Deny
Allow from all

And then all the IP-based "Deny from..." lines. Works as intended. I probably cribbed the FilesMatch bits from someone hereabouts.

[macavity]

Based on that feedback and some testing I believe there are indeed two errors here as has been suggested.

g1smd said:

Correct me if I am wrong, but I believe that only the rule in the last container will apply.

That does indeed seem to be the case :-) I've now removed that final container as it's no longer required.

lucy24 said:

the "required to enable error pages to function" reasoning doesn't sound right

You may be right, I'm not an .htaccess expert and this was added by a consultant some time ago when they were working on some efficiency gains and general improvements to what had become a very bloated file. There may be some solid reasoning behind it or there may have been a slip of the finger, I don't know, but what I do know is that removing this section and replacing it with:

<limit GET>
order allow,deny
Deny from #*$!.#*$!.#*$!.#*$!
allow from all
</limit>

seems to restore my ability to ban IPs successfully without breaking anything.

So far so good, and my sincere thanks for your help.

[macavity]

PS I've just noticed there are additional lines preceding those I've already quoted which may shed some light on the error page issue (for those who are still interested!)

# Declare custom error pages
ErrorDocument 400 /error.php
ErrorDocument 401 /error.php
ErrorDocument 403 /error.php
ErrorDocument 404 /error.php
ErrorDocument 500 /error.php

Presumably they tie in somehow with the other sections which mention error pages, i.e.

# Set up so that Allows override Denys (required to enable error pages to function)
<FilesMatch ".*">
Order deny,allow
</FilesMatch>

...


<FilesMatch "error\.(php|gif)">
Allow from all
</FilesMatch>

but I'm now very much out of my depth - just happy I've been able to ban a pesky spammer at long last!

[wilderness]

Correct me if I am wrong,

g1smd,
I've no clue.

I used one files container across a large quantity of lines for more than a decade.

[lucy24]

An error has two parts: the error code itself, and the page you serve up to people who get the error. The page is for the benefit of humans, so it should be written accordingly.

Funny illustration of the point: A robot that shall remain nameless was so used to getting locked out that it asked for the 403 error document by name. Since its last visit I had rearranged my directories, so instead of the requested "forbidden.html" it got a 404 error, leading to a "missing.html"... except that this was a locked-out robot, with no excemption for the custom 404 page, so instead it was served the 403 page. Yup, the very page it had originally asked for, just in a different location and for a different reason :)