How to explicitly allow a banned IP to view the Custom 403

It worked on Apache 1.3 but not on Apache 2.

[Angonasec]

Shared hosting, everything working as desired for years.
But... when...
I moved from Apache 1.3 to Apache 2, my zero-byte Custom 403 is no longer working as desired.

My Root .htaccess file uses mod_access to deny a list of bot IPs.

Below that I have some mod-rewrite UA banning Rules.

I scoured WebmasterWorld and tried several of jdM's suggestions, but I'm still struggling after 3 days trying.

My banning Rules all work, but in my logs it's plain from the byte-size that banned visitors are seeing the default 403: ie. Instead of 403 0 my logs show 403 298 or 403 529 etc.

Here's the relevant bits of my Root .htaccess

ErrorDocument 404 /noexist.htm
ErrorDocument 403 /403.htm

<Files *>
order deny,allow
# Nasty cidr 1
deny from nn.nnn.n.n/11
# Nasty cidr 1
deny from nn.nnn.n.n/16
# ALLOW these exceptions
allow from nn.nnn.nn.
</Files>

# FILTER REQUEST METHODS
RewriteCond %{REQUEST_METHOD} ^(OPTIONS|TRACE|DELETE|TRACK) [NC]
RewriteRule ^(.*)$ - [F,L]

# BLOCK REQUESTED URLs and QUERY STRING EXPLOITS
RewriteCond %{REQUEST_URI} \.php|\.rdf|\.asp|\.dll|register|crossdomain|\_vti\_|https?|\(null\)|proc/self/environ [NC,OR]
RewriteCond %{QUERY_STRING} (environ|iframe|localhost|mosconfig|scanner) [NC,OR]
RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]
RewriteCond %{QUERY_STRING} boot\.ini [NC,OR]
RewriteCond %{QUERY_STRING} echo.*kae [NC,OR]
RewriteCond %{QUERY_STRING} etc/passwd [NC,OR]
RewriteCond %{QUERY_STRING} \=\\%27$ [NC,OR]
RewriteCond %{QUERY_STRING} \=\\\'$ [NC,OR]
RewriteCond %{QUERY_STRING} \.\./ [NC,OR]
RewriteCond %{QUERY_STRING} \? [NC,OR]
RewriteCond %{QUERY_STRING} \: [NC,OR]
RewriteCond %{QUERY_STRING} \[ [NC,OR]
RewriteCond %{QUERY_STRING} \] [NC]
RewriteRule ^(.*)$ - [F,L]

## I added this to stop the new server looping
RewriteCond %{REQUEST_URI} !^/403.htm$
##
RewriteCond %{SERVER_PROTOCOL} ^HTTP/1\.0$ [NC,OR]
RewriteCond %{HTTP_REFERER} sites\.google|spruz|wareseeker|warez [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^$ [NC,OR]
RewriteCond %{HTTP:X-Moz} ^prefetch$ [OR]
RewriteCond %{HTTP:X-Purpose} ^instant$
## What worked on Apache 1.3
#RewriteRule .* - [F,L]
##
RewriteRule !403\.htm$ - [F]


That final Rule was jdM answer to a similar question, but it isn't working for me. My code is banning a visitor, but not allowing them to see the Custom zero-byte 403.htm file, so they get the default 403 instead.

Should I move the 403.htm to a dir outside of Root?

[phranque]

it might help to add a more specific FilesMatch container for the custom error document(s):

<FilesMatch "(403|noexist)\.htm)$">
order allow,deny
allow from all
</FilesMatch>

[g1smd]

For tidyness replace all [F,L] flags with [F].

8 of the 17 [NC] flags can be deleted as there are no characters in the pattern.

[Angonasec]

Thank you Phranque, I'm trying your suggestion by putting this at the top of my Root htaccess file just under the:

ErrorDocument 403 /403.htm

<FilesMatch "(403)\.htm)$">
order allow,deny
allow from all
</FilesMatch>

My host doesn't let me see the full access log until tomorrow, they only display a passing snippet of the error log, from which I copied this...

www.example.com [Wed Feb 26 13:53:18 2014] [error] [client 23.nn.nn.nnn] client denied by server configuration: /usr/home/mysite/public_html/example.com/403.htm
www.example.com [Wed Feb 26 13:53:18 2014] [error] [client 23.nn.nn.nnn] client denied by server configuration: /usr/home/mysite/public_html/example.com/myfile.htm

That looks to me like the bot was blocked from seeing myfile.htm, but ALSO blocked from seeing 403.htm (my Custom 403) so was not shown my zero-byte file.

In other words, the additional FilesMatch directive, didn't appear to have helped.

I suspect when I am able to download the full raw access log tomorrow, it will confirm that this hit was shown the default 403 of around 300 bytes.

[Angonasec]

Thank you g1smd; Wilco!

[Angonasec]

Updated and tidied code as advised: (Still Custom 403.htm not working)

ErrorDocument 403 /403.htm

<FilesMatch "(403)\.htm)$">
order allow,deny
allow from all
</FilesMatch>

<Files *>
order deny,allow
# Nasty cidr 1
deny from nn.nnn.n.n/11
# Nasty cidr 2
deny from nn.nnn.n.n/16
# ALLOW these exceptions
allow from nn.nnn.nn.
</Files>

# FILTER REQUEST METHODS
RewriteCond %{REQUEST_METHOD} ^(OPTIONS|TRACE|DELETE|TRACK) [NC]
RewriteRule ^(.*)$ - [F,L]

# BLOCK REQUESTED URLs and QUERY STRING EXPLOITS
RewriteCond %{REQUEST_URI} \.php|\.rdf|\.asp|\.dll|register|crossdomain|\_vti\_|https?|\(null\)|proc/self/environ [NC,OR]
RewriteCond %{QUERY_STRING} (environ|iframe|localhost|mosconfig|scanner) [NC,OR]
RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]
RewriteCond %{QUERY_STRING} boot\.ini [NC,OR]
RewriteCond %{QUERY_STRING} echo.*kae [NC,OR]
RewriteCond %{QUERY_STRING} etc/passwd [NC,OR]
RewriteCond %{QUERY_STRING} \=\\%27$ [OR]
RewriteCond %{QUERY_STRING} \=\\\'$ [OR]
RewriteCond %{QUERY_STRING} \.\./ [OR]
RewriteCond %{QUERY_STRING} \? [OR]
RewriteCond %{QUERY_STRING} \: [OR]
RewriteCond %{QUERY_STRING} \[ [OR]
RewriteCond %{QUERY_STRING} \]
RewriteRule ^(.*)$ - [F,L]

## I added this to stop the new server looping
RewriteCond %{REQUEST_URI} !^/403.htm$
##
RewriteCond %{SERVER_PROTOCOL} ^HTTP/1\.0$ [NC,OR]
RewriteCond %{HTTP_REFERER} sites\.google|spruz|wareseeker|warez [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP:X-Moz} ^prefetch$ [OR]
RewriteCond %{HTTP:X-Purpose} ^instant$
##
#RewriteRule .* - [F,L]
##
RewriteRule !403\.htm$ - [F]

[Angonasec]

PS. I did change [F,L] to [F] on the live htaccess, despite shown here in my obfuscated version :)

[Angonasec]

Out of interest, whilst waiting, here's me a few days ago, on on my previous host (Apache 1.3, spoofing as a banned Blackberry visitor in order to verify that my Custom 403.htm was indeed working properly;

nnn.nn.n.n - - [16/Feb/2014:00:10:23 -0500] "GET / HTTP/1.1" 403 - "-" "Blackberry"

Notice I was blocked and served zero bytes. :)

Exact same files on new host running Apache 2 gives only the default 403 showing in logs as; 403 298 .

[lucy24]

Edit: Oops! How did I have that tab open before starting to post?

My Root .htaccess file uses mod_access to deny a list of bot IPs.

Nope :) Now that you're in 2.x (welcome to the 21st century!) it's mod_auth-thingy.

## I added this to stop the new server looping
RewriteCond %{REQUEST_URI} !^/403.htm$

Don't make this a condition. Put a separate rule before all other RewriteRules:

RewriteRule 403\.html - [L]

The [NC] flag is meaningless except when you are matching specific alphabetic text. Even then, only use it when the thing you're testing for really can come in various casings. As a simple example, I've recently added a block on User-Agent

GoogleBot

like that. Obviously not an [NC] rule!

If you have 403s issued by more than one module-- here mod_rewrite and mod_authzzz -- then each one needs a separate exemption for the 403 page. You've got the one in mod_rewrite. Make sure there's an envelope that says

<Files "403.html">
Order Allow,Deny
Allow from all
</Files>

for mod_authz-whatsit. While you're at it, make a similar envelope for robots.txt.

[Angonasec]

Q/
mod_auth-thingy.
/Q

Ah the Cavalry's arrived! :)

Did you mean; mod_access_compat ? And has that made mod_access obsolete?

Or is it being phased/greyed out? I only ask, because my venerable mod_access Rules appear to still function, even on Apache 2. 'yer honour.

## I added this to stop the new server looping
RewriteCond %{REQUEST_URI} !^/403.htm$

Q/
Don't make this a condition. Put a separate rule before all other RewriteRules:

RewriteRule 403\.html - [L]
/Q

Thank you, I've actually deleted that line now, as the looping seems to have ceased, because of some other alteration I made. Which is progress!

[NC] Yes, I went bonkers with no-case flags a few years ago, "just to be on the safe side".
All tidy now.

Yes, I recall your exposure of that wicked imposter: GoogleBot

Q/
If you have 403s issued by more than one module-- here mod_rewrite and mod_authzzz -- then each one needs a separate exemption for the 403 page. You've got the one in mod_rewrite. Make sure there's an envelope that says

<Files "403.html">
Order Allow,Deny
Allow from all
</Files>
/Q

This sounds highly significant. But how do I implement it properly?

Being Mr. Retro, I've retained my mod_access section, and simply put this above it thus;

<Files "403.htm">
Order Allow,Deny
Allow from all
</Files>
<Files *>
order deny,allow
# Nasty cidr 1
deny from nn.nnn.n.n/11
# Nasty cidr 2
deny from nn.nnn.n.n/16
# ALLOW these exceptions
allow from nn.nnn.nn.
</Files>

I'm probably wrong eh?

Server, not looping or crashing though.
And when I spoof as a villain, I see the empty 403.htm as expected, still, but can't see if it registers in my access logs as zero-bytes or still 298 bytes.

Thank you for you help :)

[Angonasec]

Lucy Ma'am; Your suggested alterations are now live.
I'm not permitted to see my full access log until tomorrow, but this snippet just floated through my CP error window;

www.example.com [Wed Feb 26 15:43:45 2014] [error] [client 125.60.156.nnn] client denied by server configuration: /usr/home/mysite/public_html/example.com/403.htm, referer: [semalt.com...]
www.example.com [Wed Feb 26 15:43:45 2014] [error] [client 125.60.156.nnn] client denied by server configuration: /usr/home/mysite/public_html/example.com/, referer: [semalt.com...]

It's a Philippino bot.

Notice; as before it was denied access to my site homepage, the ALSO denied access to my Custom 403.htm

Do not be dismayed, this is all part of the sybaritic fun of Apache fiddling.

[lucy24]

Your suggested alterations are now live.

Yikes! Without waiting for someone to come along later and point out an egregious typo?

:: fingers crossed ::

Did you mean; mod_access_compat ? And has that made mod_access obsolete?

Or is it being phased/greyed out? I only ask, because my venerable mod_access Rules appear to still function, even on Apache 2. 'yer honour.

afaik the basic Allow/Deny rules remain the same in 2.0 and 2.2 as they were in 1.3; it's just that the relevant mod now has a different name. 2.4 is different because they're phasing out the Allow/Deny syntax. Old rules will still work, but they want you to make new ones.

An exasperating feature of error logs is that they give you all sorts of information you don't want-- but they flatly refuse to give any more detail than "client denied by server configuration". It would be nice to know which mod locked them out, wouldn't it? You can get RewriteLogs to cough up more information, but you can't use those in shared hosting. mod_security (third-party) is also helpful about telling exactly what the visitor did to offend.

as before it was denied access to my site homepage, the ALSO denied access to my Custom 403.htm

How many different mods issue 403s? I don't mean hypothetically, I mean on your specific site. Most of the time it's only mod_starts-with-a and mod_rewrite. Each one needs a separate hole poked for the custom error page. (Also possibly for things like include files, if the error page uses them.)

Double-check that you have both of these:

<Files "403.htm">
Order Allow,Deny
Allow from all
</Files>

and

RewriteRule ^403.htm - [L]

replacing "403.htm" with the name of your own error document. The Files envelope can go anywhere; put it with any similar envelopes. The RewriteRule has to go before all other RewriteRules; this is an exception to the "group rules in order of severity" principle. If your 403 document lives in a subdirectory, make sure the RewriteRule gives the full path. Files(Match) envelopes of course use only the file name.

[Angonasec]

Whilst waiting for your splendid assistance, and because the world has apparently moved on since the Siege of Mafeking; I've checked my shared server, and I'm on:

(to my alarm, notably now devoid of mod_access, and grateful for your reassurance my Deny and Allows are still acceptable in 2.2.x)

( I think I caught the typo Lucy. My Custom 403 is actually, and really, called 403.htm it's totally empty, zero bytes, and resides in the domain Root dir. )

Apache 2.2.23 FreeBSD
PHP 5.3.18
MySQL 5.1.61

Modules:
mod_authn_file
mod_authn_dbm
mod_authn_anon
mod_authn_default
mod_authz_host
mod_authz_groupfile
mod_authz_user
mod_authz_dbm
mod_authz_default
mod_auth_basic
mod_auth_digest
mod_include
mod_log_config
mod_deflate
mod_env
mod_expires
mod_headers
mod_usertrack
mod_unique_id
mod_setenvif
mod_proxy
mod_proxy_http
mod_mime
mod_status
mod_autoindex
mod_asis
mod_cgi
mod_negotiation
mod_dir
mod_imagemap
mod_actions
mod_userdir
mod_alias
mod_rewrite
mod_cband
mod_php
mod_logio


I'm currently investigating your double-check recommendations, and will report back.

I'll have yesterday's access log soon too.

Everybody's kind help is warmly appreciated :)

[Angonasec]

After much fiddling, and testing all these excellent suggestions, I've made no progress, I'm still getting the default 403, instead of banned visitors seeing my empty 403.htm (confirmed when I do get to see my raw logs the next day.)

This is what shows in the browser for a banned visitor:

Q/
Forbidden

You don't have permission to access /homepage.htm on this server.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.
/Q

Therefore, I will take another approach, and use my test domain to build an htaccess file, step by step, that does have a working custom 403

Thank you for help and interest.

For those intrigued, here's how my code looks now;

The relevant meat of my Root .htaccess file:

ErrorDocument 403 /403.htm

<Files "403.htm">
Order Allow,Deny
Allow from all
</Files>

<Files *>
order deny,allow
# Nasty cidr 1
deny from nn.nnn.n.n/11
# Nasty cidr 2
deny from nn.nnn.n.n/16
# ALLOW these exceptions
allow from nn.nnn.nn.
</Files>

Options +FollowSymLinks
RewriteEngine on

# Stop this module processing 403.htm any further
RewriteRule ^403.htm - [L]

# FILTER REQUEST METHODS
RewriteCond %{REQUEST_METHOD} ^(OPTIONS|TRACE|DELETE|TRACK) [NC]
RewriteRule ^(.*)$ - [F]

# BLOCK REQUESTED URLs and QUERY STRING EXPLOITS
RewriteCond %{REQUEST_URI} \.php|\.rdf|\.asp|\.dll|register|crossdomain|\_vti\_|https?|\(null\)|proc/self/environ [NC,OR]
RewriteCond %{QUERY_STRING} (environ|iframe|localhost|mosconfig|scanner) [NC,OR]
RewriteCond %{QUERY_STRING} (menu|mod|path|tag)\=\.?/? [NC,OR]
RewriteCond %{QUERY_STRING} boot\.ini [NC,OR]
RewriteCond %{QUERY_STRING} echo.*kae [NC,OR]
RewriteCond %{QUERY_STRING} etc/passwd [NC,OR]
RewriteCond %{QUERY_STRING} \=\\%27$ [OR]
RewriteCond %{QUERY_STRING} \=\\\'$ [OR]
RewriteCond %{QUERY_STRING} \.\./ [OR]
RewriteCond %{QUERY_STRING} \? [OR]
RewriteCond %{QUERY_STRING} \: [OR]
RewriteCond %{QUERY_STRING} \[ [OR]
RewriteCond %{QUERY_STRING} \]
RewriteRule ^(.*)$ - [F]

RewriteCond %{SERVER_PROTOCOL} ^HTTP/1\.0$ [NC,OR]
RewriteCond %{HTTP_REFERER} sites\.google|spruz|wareseeker|warez [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP:X-Moz} ^prefetch$ [OR]
RewriteCond %{HTTP:X-Purpose} ^instant$
# Belt and braces Rule to allow banned visitors to view 403.htm
RewriteRule !403\.htm$ - [F]

[lucy24]

<Files *>

What the bleep? Get rid of that envelope. Keep its contents! You don't need the envelope-- and it may be overriding the preceding one.

mod_authn_file
mod_authn_dbm
mod_authn_anon
mod_authn_default
mod_authz_host
mod_authz_groupfile
mod_authz_user
mod_authz_dbm
mod_authz_default
mod_auth_basic
mod_auth_digest

Now you understand why I am reduced to "mod_auth-thingummy".

[Angonasec]

Thank you for that.

Which, if any, is the correct way of combining them...
(check the Order Deny,Allow lines)

Number 1)

<Files "403.htm">
Order Allow,Deny
Allow from all
Order Deny,Allow
# Nasty cidr 1
deny from nn.nnn.n.n/11
# Nasty cidr 2
deny from nn.nnn.n.n/16
# ALLOW these exceptions
allow from nn.nnn.nn.
</Files>

Or ...


Number 2)

<Files "403.htm">
Order Allow,Deny
Allow from all
# Nasty cidr 1
deny from nn.nnn.n.n/11
# Nasty cidr 2
deny from nn.nnn.n.n/16
# ALLOW these exceptions
allow from nn.nnn.nn.
</Files>

Or 3) ?

Thank you for keeping at it. I'm currently building an htacc from scratch in my test domain.

But you may well have unearthed the cause of this delightful problem.

Will I have it fixed by the weekend, I wonder?

[Angonasec]

I think it is 3)

<Files "403.htm">
Order Allow,Deny
Allow from all
</Files>

# Nasty cidr 1
deny from nn.nnn.n.n/11
# Nasty cidr 2
deny from nn.nnn.n.n/16
# ALLOW these exceptions
allow from nn.nnn.nn.

Am I correct?

[Angonasec]

It appears to be working at last. I'll have to check my raw access logs tomorrow to be certain. But so far so good, on the live site.

My new Host tells me that my logs will not display a zero byte file as zero bytes.... because
Q/
It's likely the old host was using a different logging format. Under Apache, if you use the %b variable for the size definition in the access log, that will list just the returned object size and not include the response headers. That is likely how your previous host had configured their logs. At this network, we use %O instead, which will return the true size of what was sent back.

That does include the server response headers from it.

For any response from the server, that will include the server headers, so that field won't ever go to 0. The headers returned can be viewed via the wget command or any other tool that displays response time headers.
/Q

So the bytes I see for my empty 403.htm will be from server headers.

Thank you all, especially Lucy, for your splendid assistance I owe you Raspberry Ripples.

[lucy24]

I think it is 3)

I think your server will explode if you have more than one "order" statement in the same
:: racking brains ::
scope.

Yes: You make a separate <Files> envelope for just the files that require special treatment. In addition to the one for 403.htm, you've probably got one for robots.txt. I also have one for favicon.ico and a FilesMatch for css. Both of those are to help identify wrongly blocked humans. (It also and coincidentally meant that the old faviconbot, the one with a blank UA, was still able to do its thing.)

My new Host tells me that my logs will not display a zero byte file as zero bytes.... because

That makes sense. I've seen other places where different people report different sizes on what ought to be the identical response. It's the extra stuff in the headers.

Do you really return an empty 403? That's not very nice to the stray humans who mistakenly request a directory that has no index file. And those are, in practice, the only people who will ever really look at the 403 document.

[Angonasec]

Fear not fair Maid, I didn't actually use that bizarre line; pondering the logic I guessed the correct implementation.

Q/
In addition to the one for 403.htm, you've probably got one for robots.txt. I also have one for favicon.ico and a FilesMatch for css. Both of those are to help identify wrongly blocked humans. (It also and coincidentally meant that the old faviconbot, the one with a blank UA, was still able to do its thing.)
/Q

On balance, considering the parlous state of the modern internet, our policy is to deny access to robots.txt to all except pre-approved visitors. Debated ad-inf elsewhere.

Q/
Do you really return an empty 403? That's not very nice to the stray humans who mistakenly request a directory that has no index file. And those are, in practice, the only people who will ever really look at the 403 document.
/Q

As empty as I can make it. I resent blocked traffic even getting the server header data.

Let them eat vacuum!

We have in place (non-htaccess) mechanisms to direct "stray" humans to a trout-ladder.

About which, we must remain enigmatic for obvious reasons :)

Signing off here: Thank you again.

[lucy24]

I resent blocked traffic even getting the server header data.

You could quietly rewrite all unwanted page requests to a one-pixel gif. It doesn't have the visceral satisfaction of a 403, but absolutely nothing creates a smaller response. On my system it comes through as 352 bytes, of which 85 are the file itself. On the same system, a redirect-- such as the occasionally pleasing redirect to 128.0.0.1 -- runs around 5-600 bytes. (No, I don't know why there's such a range. My filenames certainly don't vary in length by 100+ characters.) And 403s run around 2900.

At this point I detoured to figure out how big my 403 page is-- the html as such is about 1600, but it's got a couple of includes-- and am now stumped, because the page appears to be bigger than what the server sends out.

I am also at a loss to understand why 403s on one site are almost twice as big as on the other, even though the pages are identical and the included stuff can't possibly differ by more than 100 bytes or so.

Huh. This calls for further investigation.

Well, I had to stay up anyway. Bread flatly refused to rise for several hours.

[Angonasec]

Keep talking, I'm listening Ma'am :)

I'm not after "satisfaction", I'm trying to stop wasted BW and CPU. 50% of hits to our site this week were unwanted traffic; it mounts up rapidly.

Here's what our live site served to another East German OVH vandal once I'd fixed the Custom 403.htm problem... they all get around 294 bytes when presented with the zero-byte 403.htm (courtesy of Ms Lucy's assistance).

94.23.161.188 - - [27/Feb/2014] "GET /myfile.htm HTTP/1.1" 403 294 "-" "Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/20100101 Firefox/6.0"