Blocking IP ranges

What is the correct way in apache 2.4?

[dstiles]

Basically, trying to create a blacklist of IP ranges. I've found several "do it like this" items (mostly the same) but none seem to block.

Sample VirtualHost...

<VirtualHost xxx.xx.xxx.xxx:443>
 ServerAdmin alert@maildomain
 ServerName www.example.com
 DocumentRoot /srv/example
 <Directory "/">
  AllowOverride None
  Require all denied
 </Directory>
 <Directory "/srv/example">
  DirectoryIndex index.php
  AllowOverride All
  Include /etc/apache2/use-setenv.conf
  Include /etc/apache2/rewrite.conf
  <RequireAll>
   Require all granted
   Include /etc/apache2/ban-ips-not.conf
  </RequireAll>
 </Directory>
(etc)

ban-ips-not.conf

Require not ip 3.0.0.0/8
Require not ip 34.192.0.0/10
Require not ip 54.64.0.0/8
(etc)

I've tried re-ordering the includes to no effect. The setenv and rewrite includes both function as expected.

I've also tried...

  Include /etc/apache2/use-setenv.conf
  Include /etc/apache2/rewrite.conf
  Include /etc/apache2/ban-ips.conf

with the include file ban-ips.conf...

<RequireAll>
 Require all granted
 <RequireNone>
  Require ip 3.0.0.0/8
  Require ip 34.192.0.0/10
  Require ip 54.64.0.0/8
  (etc)
 </RequireNone>
</RequireAll>

Most solutions I've seen use the former format. Neither works for me. Both reload with no errors but neither blocks the ip ranges.

Anyone spot what I'm doing wrong, please?

[lucy24]

I've also tried...

This makes it sound as if you mean that the lines simply aren't getting included--which is an entirely different issue from the <RequireNone> vs. Require not option. Have you verified that the rule works if you manually type in the Require lines instead of using Include?

Are other files getting successfully included?

with the include file ban-ips.conf...

Did something get left out in editing? Now it sounds as if the file ends up being

<RequireAll>
 Require all granted
 <RequireAll>
  Require all granted
  <RequireNone>
  Require ip 3.0.0.0/8
  Require ip 34.192.0.0/10
  Require ip 54.64.0.0/8
  (etc)
  </RequireNone>
 </RequireAll>
</RequireAll>

which can't possibly be what you meant to say.

:: detour to docs to verify that syntax of Include statements hasn't changed significantly between 2.2 and 2.4 ::

[dstiles]

Thanks for the reply, Lucy.

# <RequireAll> - sorry, I should have included more code to make it clear. in virtualhost it should be exactly as shown with no Requires intervening.

The other two files shown work fine. I've tried the ip-ban file both above and below them, with no difference.

The lines seem to be included - a mistake in the file causes a reload error. Copying a few IPs to the example.conf file, from the first code section above...

 <RequireAll>
 Require all granted
# Include /etc/apache2/ban-ips-not.conf
 Require not ip (my ip)
 Require not ip 3.0.0.0/8
 Require not ip 34.192.0.0/10
 Require not ip 54.64.0.0/8
 </RequireAll>

...does not block my ip. Also tried, with no blocking...

 <RequireAll>
 <RequireNone>
  Require all granted
# Include /etc/apache2/ban-ips-not.conf
  Require not ip (my ip)
  Require not ip 3.0.0.0/8
  Require not ip 34.192.0.0/10
  Require not ip 54.64.0.0/8
 </RequireNone>
 </RequireAll>

Omitting Require all granted causes (as expected) a reload error.

Can't find my ip anywhere else that matters. I've also checked htaccess but that only holds security headers.

[lucy24]

Omitting Require all granted causes (as expected) a reload error.

Heehee, yes, when my server moved to 2.4 earlier this year, it took me a while to figure out that the containing <RequireAll> (or perhaps <RequireAny>, didn't have occasion to try) is obligatory; any <RequireNone> has to go inside that.

<RequireAll>
<RequireNone>
Require all granted

Cut-and-paste error? I'd expect the “Require all granted” to be inside the RequireAll, outside the RequireNone. Same for all those "Require not" lines inside RequireNone--wouldn't they all cancel each other out?

My current htaccess, which works, says in part

<RequireAll>
Require all granted
<RequireNone>
Require env keep_out
Require env bad_range
Require env noagent
Require env bad_agent
[ snip ]
Require ip 195.181.aa.bb
</RequireNone>
</RequireAll>

I don't currently have any rules pertaining to IP ranges--the “Require ip” was for a temporarily vexatious robot that has since gone away--though I do use the bad_range environmental variable for several of The Usual Suspects. (It has to be done that way so I can un-set it for distributed robots.)

:: quick detour to logs to verify that the most recent visits of 195.181.aa.bb were in fact blocked ::

[dstiles]

# Cut-and-paste error
Umm... Yes. Sorry.

I have ensured my eample.conf has the same format as your example (without the env's) and it still does not block. I must have something dumb set up but just can't see it.

[lucy24]

Hmmm....

:: scratching head in exasperation ::

Start with this. In some remote corner of the site, in a directory where htaccess is permitted, put in an htaccess that says simply

<RequireAll>
Require all granted
Require not ip aa.bb.cc.dd
</RequireAll>

giving your own exact IP address, and verify that it works.

Now change “aa.bb.cc.dd” to “aa.bb.cc.0/24” and verify that it still works. We will continue from there. Here I’m recommending htaccess because you can just make changes on the fly without reloading the whole config file.

Oh! Also, do a global search of ALL your configuration files and make sure the words “Allow from” or “Deny from” or “Order” do not occur anywhere. (I also learned by sad experience that Apache isn’t kidding when it says that mod_auth-whatever-it-is--the one with Require--in conjunction with mod_compat-thingy--the one that handles 2.2-style directives--leads to the dreaded “unintended consequences”.)

[dstiles]

Thanks, Lucy. That works. And no old-style Allow/etc.

Putting the same code in example.conf passes configtest but does not block single IP. However...

Commenting out the other incudes DOES block the IP. Whether the enabled includes are above or below the Require not ip makes no difference. With the includes enabled the ban in example.conf does not block, in htaccess it does.

Further playing around: commenting out the setenv include with the test code enabled blocks the IP (ie the rewrite include is still active). With setenv include enabled, either above or below the test code, the block fails. Conclusion: something in setenv is over-riding the test code. The rewrite one is ok.

Following this, there is a chunk of Requires at the end of the setenv file that control bot access. Removing this, the test ban works. So I'm looking at...

example.conf:

<VirtualHost xxx.xx.xxx.xxx:443>
 ServerAdmin alert@mailserver
 ServerName www.example.com
 DocumentRoot /srv/example
 <Directory "/">
  AllowOverride None
  Require all denied
 </Directory>
 <Directory "/srv/example">
  DirectoryIndex index.php
  AllowOverride All
  Include /etc/apache2/use-setenv.conf
  Include /etc/apache2/rewrite.conf
  <RequireAll>
   Require all granted
   Require not ip (my ip)
  </RequireAll>
 </Directory>
 (etc...)
</VirtualHost>

use-setenv.conf (final part):

Require env dave_is_good
Require env good_uri
Require env robots_txt_uri
<RequireAll>
 # allow bad googlebot parms
 Require env google_bot
 Require env google_bot_ip
 Require env old_browser
</RequireAll>
<RequireAll>
 # allow duckduck and cliqz
 Require env amazon_ips
 <RequireAny>
  Require env cliqz
  Require env duck
 </RequireAny>
</RequireAll>
<RequireAll>
 Require method GET POST HEAD
 <RequireNone>
  Require env too_low_proto
  Require env bad_proto
  Require env noaccept
  Require env noaccept_lang
  (etc... - all Require env)
 <RequireNone>
</RequireAll>

The googlebot and duck/cliqz sections have been added since the original probelm so can be discounted (I also removed them to check). The get/post/head is included because it has to be somewhere and a positive Require is needed there. There are about 40 Require env lines.

I have tried removing parts of the above, including all the None section and each and all of the exception sections. Nothing works except complete removal of the entire section, which suggests I have an error in it somewhere.

[lucy24]

use-setenv.conf (final part):

Does the whole thing go inside a <RequireAll> envelope? Or a <RequireAny>? A <RequireNone>? I can't figure out what's going on in the opening lines. And why so many separate <RequireAll> envelopes?

Tangent:

Require env robots_txt_uri

If the idea here is to let everyone see robots.txt, you can also do it with a <Files> envelope containing the single line "Require all granted". Which reminds me: make sure you’ve also got something that allows everyone to see your custom 403 page, assuming you've got one.

[dstiles]

There are no Requires in use-setenv.conf before the ones I've noted and none in the virtualhost other than those noted. The rewrite.conf file contains only rewritecond/rewriterule lines. and the htaccess, as I said, contains only header set commands.

The Require All blocks at the start are to enable googlebot, which uses old browsers, and cliqz/duckduckgo which use some ranges of Amazon IPs (currently listed in a single setenvif earlier in the config). I've tried removing the first part of the conf, which comprises setenv/browsermatch and a single <if -R... || -R...> which matches IP ranges for facebookexternalhit but no effect.

Of course, the code I tried in htaccess is run after everything else. If I could include the bad IPs in htaccess that would probably solve the problem, but still wouldn't explain why this doesn't work.