IP Deny Question

[EarleyGirl]

I have posted this elsewhere but should have tried here first. In my .htaccess file I am denying the following IP blocks:
order allow,deny
deny from 78.0.0.0/8
deny from 88.0.0.0/8
allow from all

Am I correct in assuming that the above should successfully block all IPs within the ranges below?
78.0.0.0 to 78.255.255.255
88.0.0.0 to 88.255.255.255

Here are two recent access log entries:
88.198.112.187 - - [12/Jan/2009:02:30:42 -0800] "GET / HTTP/1.1" 301 568 "spam-domain-here/freescripts/my-domain-here/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16"

78.47.30.13 - - [09/Jan/2009:03:29:29 -0800] "GET / HTTP/1.1" 301 568 "spam-domain-here/bestof/my-domain-here/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16"

While I'm not an expert, I have been denying IPs with .htaccess for years. I know this .htaccess file works as I've tested using my own IP with no problem. Yet the above IPs have successfully accessed the site. Where have I gone wrong?

Thanks in advance for helping me solve the mystery.

[wilderness]

Am I correct in assuming that the above should successfully block all IPs within the ranges below?
78.0.0.0 to 78.255.255.255
88.0.0.0 to 88.255.255.255

Yes.

Perhaps there's something missing within your htaccess which you've not provided?

Note, both of your examples include the following:

/ HTTP/1.1" 301

A normal access would read / HTTP/1.1" 200

What may be happening in these instances are the visitors are requesting your root directory, which is absent a trailing slash (a common trick for some denied strategists) to confirm a presence.

[Samizdata]

Perhaps you have a "canonical redirect" (non-www to www) that produces the 301 response?

...

[EarleyGirl]

Thanks wilderness. Yes, I have a lot of IPs denied in that file and did have a rewrite from non www to www.domain.com. I've removed that since the host provides this via the CP. Now I show HTTP/1.1" 304 when I access.

I denied my IP and then tried to access mydomain.com without the trailing slash but was denied access anyway. Any other ideas? I'd hate to post the whole .htaccess file.

[EarleyGirl]

Perhaps you have a "canonical redirect" (non-www to www) that produces the 301 response?

Yes... I did and that was what I meant to say above but I blanked. Thanks Samizdata!

[Samizdata]

I don't think you have anything to worry about.

The 304 response just means that the content hadn't changed since your browser last loaded it.

The 301 is a permanent external redirect - when the request is made the server says "no soup for you, go to this address instead" - normally it will be a full URL on your site starting with the "http" (such as your www domain root) but it could be to another server altogether if that is how you have it configured.

Some bots don't follow such redirects. Those that do will be denied access by your code:

"GET / HTTP/1.1" 301 310
"GET / HTTP/1.1" 403 -

I am not aware of any downside to this approach (but if there is someone will soon say so).

...

[EarleyGirl]

Thanks, Samizdata. I'm not sure I understand because I've removed the line for rewriting to the www address now so I'm guessing it doesn't apply.

If it helps any, I have all mod_rewrite listed below the deny section.

I do have the "order allow,deny" set correctly (as far as that line goes), don't I? I mean, it is working as far as denying a lot of IPs. I just didn't understand why it wasn't working for those two.

Thanks again.

[jdMorgan]

Two suggestions/guesses:

Be aware that if you have multiple unconditional "Order" directives in a .htaccess file, only the last one found will be used. By "unconditional" I mean "Located outside of <Files> or <Limit> containers which are not mutually-exclusive."

Be sure your custom ErrorDocuments (if you use any) are declared as local filepaths only -- Do not include "http://example.com" in the path. They should look like this:

ErrorDocument 403 /local-path-to-403-error-page

Jim

[EarleyGirl]

Thanks jdMorgan. It is the last one as far as the "Order" directives and the ones prior are conditional.

I do not use custom ErrorDocuments for this site.

I did find an error in one of the rewrite rules but I'm not sure how would cause some IPs to slip through and others not.

Anyway, thanks again.

[Caterham]

have a rewrite from non www to www.domain.com. I've removed that since the host provides this via the CP.

And how did your host setup that? Is that implemented in the translate_name phase?

I did find an error in one of the rewrite rules but I'm not sure how would cause some IPs to slip through and others not.

Not possible. Unless you configured that the filename bypasses or passes the access check mod_rewrite's fixup hook can't be reached.

[EarleyGirl]

And how did your host setup that? Is that implemented in the translate_name phase?

Thanks, Caterham. Sorry... I don't know how the host set it up. I do know it's not through the .htaccess file.

[EarleyGirl]

I wanted to mention that someone else advised that I should use only the first two numbers rather than what I've used. So instead of deny from 78.0.0.0/8, I should use deny from 78 as 78.0.0.0/8 either does not include the entire range of 78.0.0.0 to 78.255.255.255 or is unnecessary. I thought the opposite was true. Maybe someone here would clarify that for me? Perhaps I'm not using the correct format for denying the entire range?

[jdMorgan]

Either format is acceptable. See the Apache mod_access documentation for the various Allow and Deny IP address and range format.

All of these are equivalent and valid:

Deny from 78.0.0.0/8
Deny from 78.0.0.0/255.0.0.0
Deny from 78

The first uses "CIDR" notatation, the second uses a netmask, and the last one uses a partial (prefix) address. You can use whichever you like (even mixing them) and it should make no difference.

It is the last one as far as the "Order" directives and the ones prior are conditional.

As I said, *all* must be mutually-exclusive. As you describe it, the "ones prior" which are conditional will be overridden by the last (unconditional) Order directive. This may not be the cause of your trouble, but is worth some care.

I would be looking for a redirect that executes prior to your .htaccess code -- most likely in the server configuration as Caterham pointed out above. The more control you want in your .htaccess file, the fewer "control panel" functions you should use, as they will execute first.

This can impact access controls as seen here, or cause multiple "chained" or "stacked" redirects when you try to canonicalize URLs, such as when redirecting "http://<any-domain>/index.php" to http://<canonical-domain>/".

For example, although your .htaccess code will normally redirect any "wrong domain" to the right domain at the same time as redirecting "/index.php" to "/", you will always see two redirects if /index.php is requested with a non-canonical hostname and a "control panel domain redirect" is enabled; First the "control panel" domain redirect will be invoked, redirecting the wrong domain to the canonical domain, and then your .htaccess "/index.php"-to-"/" redirect will be invoked. The result is two chained redirects, with less likelihood that PageRank/link-popularity will be passed through these chained redirects -- or will be passed through in a timely manner.

Jim

[EarleyGirl]

Thanks, Jim. From now on, I will control redirects with .htaccess rather than the CP as I definitely had some chained redirects going on. Could this affect the IP denial area though?

This is what I have above the allow/deny:
<Files ~ ".htaccess">
Order allow,deny
Deny from all
</Files>

Then I have several - SetEnvIf User-Agent (various) getout
then my order allow,deny (IPs denial) section begins.
I take it from your explanation that the <Files ~ ".htaccess"> code is overridden by the last section, causing it to be ineffective, correct?

[wilderness]

Deny from 78

I'm sure Jim hadn't reached his sixth pot of coffee yet, however, in order for the above to function a trailing period is necessary.

Deny from 78.

[jdMorgan]

Again, the authoritative source is Apache: [httpd.apache.org...]

Jim

[EarleyGirl]

Thanks everyone for all the help!