is deny from . cached?

[Frank_Rizzo]

A few times now I've had this scenario:

An IP address or cidr range is blocked due to suspicious activity. Entry added to httpd.conf file and apache restarted.

But this was a false positive and a genuine visitor to the site is now getting 403'd on every page.

I go back to the httpd.conf file and remove the false positive address. Apache restarted but the user is still getting 403's! Not only that but I now reboot the server and the 403 is still being served to that particular user.

The only thing I can do is to delete a whole chunk of older entries and restart apache.

I don't know if this is somekind of caching but I doubt that as the server is rebooted. Maybe it's to do with the size of the deny from... list. I did ask on a different thread if it's not a good idea to ban like hundreds / a thousand or so in one list.

Is there anyother way a user can be blocked which I'm missing out here? I got this visitor and he's been locked out for 3 hours. His IP was banned this morning, I took that ban out an hour later, but still now he can't get in.

-----

Note I did post a similar thread here:

[webmasterworld.com...]

and thought I had resolved this issue but it has come back again.

[edited by: Frank_Rizzo at 2:18 pm (utc) on April 8, 2007]

[edited by: jdMorgan at 2:52 pm (utc) on April 8, 2007]
[edit reason] linked previous thread [/edit]

[jdMorgan]

> Is the [403 page] cached?

Yes it is, unless you take steps to make it uncacheable, it is cached in the client's browser. It is cached as the URL that the client attempted to request, and not under the 403 page page URL.

If you have a custom 403 page named, as an example, custom403.html, you might want to give it a short or even instant expiry date.

Here's a longish example showing an Expires and Cache-Control 'environment':

# Set up Cache Control headers
ExpiresActive On
#
# Default - Set http header to expire everything 1 day from last access
ExpiresDefault A86400
#
# Apply customized Cache-Control and expires headers to custom 403 file
<FilesMatch "^custom403\.html$">
ExpiresDefault A0
Header set Cache-Control: "no-store"
# Disconnect client immediately after 403 response
SetEnv nokeepalive
</FilesMatch>

Note that these controls are applied per-FILE, not per-page or per-URL. This is necessary because of the client-side caching behaviour described above. See Apache mod_expires and mod_headers

Jim

[Frank_Rizzo]

I think this is something more than caching. It's just not right here.

Here's some entries from this morning:

-----------------
[Tue Apr 10 08:18:29 2007] [error] [client 89.53.abc.abc] client denied by server configuration: /home/widgets/public_html/
[Tue Apr 10 08:49:48 2007] [error] [client 85.179.abc.abc] client denied by server configuration: /home/widgets/public_html/, referer: http : // www.google.de/search?q=widgets&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:de:official&client=firefox-a
[Tue Apr 10 08:50:06 2007] [error] [client 85.179.abc.abc] client denied by server configuration: /home/widgets/public_html/
[Tue Apr 10 08:50:07 2007] [error] [client 85.179.abc.abc] client denied by server configuration: /home/widgets/public_html/
[Tue Apr 10 09:41:48 2007] [error] [client 210.49.abc.abc] client denied by server configuration: /home/widgets/public_html/widgetsforum/viewtopic.php, referer: http : // www.google.com.au/search?q=widgets&hl=en&rlz=1T4HPAA_en___AU204&start=10&sa=N
[Tue Apr 10 09:41:49 2007] [error] [client 210.49.abc.abc] client denied by server configuration: /home/widgets/public_html/favicon.ico
[Tue Apr 10 10:07:56 2007] [error] [client 220.239.abc.abc] client denied by server configuration: /home/widgets/public_html/widgets.php
[Tue Apr 10 10:22:43 2007] [error] [client 210.49.abc.abc] client denied by server configuration: /home/widgets/public_html/, referer: http : // www.someotherwidgetsite.com/home.htm
-----------------
(obviously changed true IP to .abc.abc)

Those IP addresses are not in my deny file. Neither is the cidr range. Here is a short extract of the deny file which is included in httpd.conf:

---------------------------
#ban_ips.conf

<Directory "/home">
<Files ~ "^.*$">
order allow,deny
allow from all

#10-Apr-07
deny from 200.88.192.0/18 #do
deny from 24.121.0.145 #npg
deny from 61.17.38.45 #in
deny from 211.129.0.0/16 #jp
deny from 201.143.0.0/18 #mx
deny from 58.88.0.0/15 #jp
deny from 218.145.26.168 #kr

#09-Apr-07
deny from 65.118.34.29 #qwest
deny from 202.56.192.0/18 #india
deny from 203.131.181.190 #ph
deny from 64.92.199.36 #data393
deny from 210.4.0.0/21 #ph
deny from 216.255.64.0/19 #digi md
deny from 65.40.0.0/15 #embar fl
deny from 212.241.179.54 #donhost
deny from 200.41.39.120/29 #ar
deny from 84.47.0.0/17 #slovakia
deny from 68.32.0.0/11 #comcast
deny from 69.60.118.55 #infolink
.
.
.
</Files>
</Directory>
------------------------

(not changed IP's as they are bad guys). This file contains a list of recent suspicious activity IP's which are banned by myself. At the bottom of the file is a list of complete country IP's.

Note that this file is currently 2189 lines long and that is what I suspect could be the problem.

For some reason genuine visitors are being 403'd even though the IP and / or cidr are not listed in this file.

Take the 210.49 example at 10:22:43. If I search the file for 210.49 there is no entry! If I widen the search for 210.4 I get the following lines:

deny from 210.4.0.0/21 #ph
deny from 210.4.3.66 #ph
deny from 210.40.0.0/13 #cn

So would any of the above ban 219.49.abc.abc?

A better example is the 89.53 at 08:18:29

I search the ban_ips.conf file for

89.5

and nothing is found!

So what procedure is serving a 403 to ip address 89.53.abc.abc?

Here is the access log file for the full 89.53.abc.abc address for this month:

---------------------------
89.53.abc.abc - - [10/Apr/2007:08:14:54 +0100] "GET /widgetsforum/viewtopic.php?p=10430 HTTP/1.1" 403 - "http: // www.google.de/search?hl=de&q=widgets+&btnG=Google-Suche&meta=" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" 0 widgets.com "-" "-"
89.53.abc.abc - - [10/Apr/2007:08:15:06 +0100] "GET /widgetsforum/viewtopic.php?p=10430 HTTP/1.1" 403 - "http : // www.google.de/search?hl=de&q=widgets+&btnG=Google-Suche&meta=" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" 0 widgets.com "-" "-"
89.53.abc.abc - - [10/Apr/2007:08:15:06 +0100] "GET /favicon.ico HTTP/1.1" 403 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" 0 widgets.com "-" "-"
89.53.abc.abc - - [10/Apr/2007:08:18:25 +0100] "GET /widgetsforum/viewtopic.php?p=10430 HTTP/1.1" 403 - "http : // www.google.de/search?hl=de&q=widgets+&btnG=Google-Suche&meta=" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" 0 widgets.com "-" "-"
89.53.abc.abc - - [10/Apr/2007:08:18:29 +0100] "GET / HTTP/1.1" 403 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3" 0 widgets.com "-" "-"
------------------------

Why were these 5 accesses all served with a 403?

[edited by: Frank_Rizzo at 10:02 am (utc) on April 10, 2007]

[jdMorgan]

Take the 210.49 example at 10:22:43. If I search the file for 210.49 there is no entry! If I widen the search for 210.4 I get the following lines:
...
deny from 210.40.0.0/13 #cn

This CIDR is the 'closest' one, but denies only 210.40.0.0 through 210.47.255.255, so obviously it's not applying here. Combined with your other reported inconsistencies, there are only a couple of things I'd suggest:

1) Comment-out *all* of the mod_alias code for a day or two, and see if you still get any strange IP range denials. This would indicate a problem in another section of code, or even in a different .htaccess or server config file.

2) Examine your code for other access-control code: Directives which deny by remote address, remote host, user-agent, referrer, requested URI, or query-string are all prime candidates. In addition, you may have code that examines proxy-related request headers, and any of these could be responsible for the denials. Look for problems there, too, and comment-out large sections until you no longer get unexplainable access denials.

If you deny by remote-host, be aware that a temporary failure or timeout in reverse-DNS lookups will result in a blank remote-hostname, and you must be sure that your code can handle this possibility. Otherwise, your server is depending upon a lot of additional hardware and software in the DNS system to work.

The size of your file is not a problem -- I've seen much larger .htaccess files, including some of my own.

It is not recommended to put comments on the same line as directive; This will throw a "Warning" error, which depending on the server config, may or may not be logged. Comments should be on a separate lines starting with "#". That is not the main problem here, though.

Jim

[Frank_Rizzo]

Thanks Jim, some good testing advice there.

First task - move the # comments.