My .htaccess blocking IPs that are not entered - bizarre-oh! - Apache Web Server forum at WebmasterWorld

Forum Moderators: phranque

Message Too Old, No Replies

My .htaccess blocking IPs that are not entered - bizarre-oh!

erlandc

1:20 am on Aug 14, 2014 (gmt 0)

Hi all,
I'm no expert but my friend cannot access one of my sites. I see his IP address in my logs but it's not in my htaccess file, plus another also
Where have I gone wrong? (I'm pulling out my hair)
thx
e

not2easy

1:44 am on Aug 14, 2014 (gmt 0)

Are you only blocking via IP?
Is your friend seeing an error page or what?
Where you see the IP of your friend, can you also see the server response?

erlandc

1:49 am on Aug 14, 2014 (gmt 0)

(via IP & RewriteEngine) he sees 403 Forbidden - I see it in my log files. (not sure what you mean by server response) thx

not2easy

1:59 am on Aug 14, 2014 (gmt 0)

Server response is the number used by a server to tell you how it handled a request. A 200 means all OK, a 403 means access forbidden.

Can you paste the lines from your htaccess (replacing your private information with "example.com" if needed) so people can help you find the problem? Without that, it is anybody's guess, sorry.

erlandc

2:05 am on Aug 14, 2014 (gmt 0)

ok thx, it was 403 forbidden

you want all of my bloated htaccess file?

erlandc

2:12 am on Aug 14, 2014 (gmt 0)

here is is - I actually learned from here...

<Files *>
order allow,deny
deny from 8.18.144.0/23
deny from 23.20.0.0/14
deny from 46.51.215.0/25
deny from 46.51.215.128/26
deny from 46.51.215.192/27
deny from 46.51.215.224/28
deny from 46.51.215.240/29
deny from 46.51.215.248/30
deny from 46.51.215.252/31
deny from 46.51.215.254/32
deny from 46.51.128.0/18
deny from 46.51.192.0/20
deny from 46.51.208.0/22
deny from 46.51.212.0/23
deny from 46.51.214.0/24
deny from 46.51.216.0/21
deny from 46.51.224.0/20
deny from 46.137.0.0/16
deny from 54.224.0.0/11
deny from 63.92.12.0/22
deny from 63.238.12.0/22
deny from 63.238.16.0/23
deny from 66.7.64.0/19
deny from 67.202.0.0/18
deny from 67.205.69.32/27
deny from 72.21.192.0/19
deny from 72.29.185.0/24
deny from 72.44.32.0/19
deny from 72.55.128.0/18
deny from 79.125.0.0/16
deny from 81.144.138.34
deny from 81.144.138.40
deny from 87.231.235.2/32
deny from 107.20.0.0/14
deny from 174.129.0.0/16
deny from 184.72.0.0/15
deny from 204.74.108.0/24
deny from 204.236.128.0/17
deny from 204.246.160.0/22
deny from 204.246.167.0/24
deny from 204.246.168.0/23
deny from 204.246.176.0/21
deny from 204.246.184.0/22
deny from 207.171.160.0/19
deny from 208.47.248.0/23
deny from 209.201.96.0/22
deny from 216.137.32.0/20
deny from 216.137.48.0/21
deny from 216.182.224.0/20
deny from 5.0.0.0/8
deny from 5.135.125.179
deny from 5.135.125.182
deny from 5.135.125.183
deny from 64.213.218.112
deny from 1.0.0.0/8
deny from 69.63.176.0/20
deny from 24.125.192.215
deny from 64.233.160.0/19
deny from 70.29.145.119
deny from 70.29.148.187
deny from 204.15.80.50
deny from 204.15.80.0/22
deny from 58.0.0.0/8
deny from 198.103.53.0/24
deny from 67.202.0.0/18
deny from 72.44.32.0/19
deny from 79.125.0.0/17
deny from 174.129.0.0/16
deny from 204.236.128.0/17
deny from 204.236.224.0/19
deny from 23.22.163.55
deny from 24.4.111.95
deny from 24.58.0.0/15
deny from 24.59.94.137
deny from 24.125.202.29
deny from 24.136.95.151
deny from 24.168.35.217
deny from 24.232.0.0/16
deny from 31.0.0.0/8
deny from 37.0.0.0/8
deny from 38.0.0.0/8
deny from 38.113.234.181
deny from 41.0.0.0/8
deny from 23.20.0.0/14
deny from 54.72.0.0/13
deny from 54.80.0.0/12
deny from 67.202.0.0/18
deny from 184.72.0.0/15
deny from 204.236.128.0/17
deny from 107.20.0.0/14
deny from 54.80.214.201
deny from 54.176.0.0/12
deny from 54.204.0.0/15
deny from 54.221.0.0/16
deny from 54.224.0.0/12
deny from 54.234.0.0/15
deny from 54.240.0.0/12
deny from 59.95.210.66
deny from 60.0.0.0/8
deny from 60.48.69.4
deny from 60.50.61.192
deny from 60.240.248.19
deny from 60.240.248.23
deny from 60.240.248.24
deny from 60.191.80.45
deny from 61.247.219.44
deny from 61.247.219.50
deny from 63.209.222.18
deny from 63.251.92.200
deny from 64.111.127.81
deny from 64.124.0.0/15
deny from 64.125.222.16
deny from 164.138.209.245
deny from 64.211.112.0/21
deny from 64.211.0.0/17
deny from 64.211.128.0/18
deny from 64.211.192.0/19
deny from 64.229.86.42
deny from 64.235.144.0/20
deny from 65.28.108.199
deny from 65.52.0.0/14
deny from 65.78.161.42
deny from 65.92.28.37
deny from 65.92.216.70
deny from 65.98.224.3
deny from 65.210.123.237
deny from 65.214.39.180
deny from 66.34.204.26
deny from 66.45.252.90
deny from 66.90.118.101
deny from 66.96.128.0/18
deny from 66.111.32.0/19
deny from 66.120.0.0/13
deny from 66.130.218.119
deny from 66.136.0.0/13
deny from 66.147.244.213
deny from 66.150.0.0/15
deny from 66.150.224.245
deny from 66.175.164.24
deny from 66.182.88.94
deny from 66.201.230.0/24
deny from 66.212.253.254
deny from 66.212.254.217
deny from 66.220.144.0/20
deny from 66.228.210.10
deny from 66.228.210.50
deny from 66.228.202.22
deny from 66.232.111.94
deny from 67.15.0.0/16
deny from 67.55.72.165
deny from 67.70.153.23
deny from 67.78.34.170
deny from 67.86.56.104
deny from 67.202.56.158
deny from 67.202.0.0/18
deny from 67.202.24.96
deny from 67.202.49.33
deny from 67.168.154.188
deny from 67.202.25.192
deny from 67.205.96.152
deny from 67.215.234.250
deny from 67.228.100.141
deny from 69.9.181.169
deny from 69.11.119.107
deny from 69.25.142.42
deny from 69.36.144.0/20
deny from 69.36.158.36
deny from 69.37.0.0/16
deny from 69.37.32.0/23
deny from 69.58.176.0/20
deny from 69.58.178.29
deny from 69.64.145.225
deny from 69.90.103.173
deny from 69.149.107.70
deny from 69.162.128.0/18
deny from 69.171.224.0/19
deny from 69.211.0.0/20
deny from 70.33.160.0/19
deny from 70.33.184.32/27
deny from 70.42.37.7
deny from 70.42.186.129
deny from 70.48.219.155
deny from 70.84.0.0/14
deny from 70.85.81.76
deny from 70.87.15.50
deny from 70.113.80.43
deny from 70.168.200.180
deny from 70.176.70.77
deny from 70.240.0.0/12
deny from 71.13.115.117
deny from 71.71.7.51
deny from 71.197.96.124
deny from 71.202.29.212
deny from 71.204.21.71
deny from 71.232.218.119
deny from 72.8.151.0/24
deny from 72.8.191.0/24
deny from 72.13.32.0/19
deny from 72.36.180.234
deny from 72.36.64.0/18
deny from 72.44.32.0/19
deny from 72.44.58.248
deny from 72.44.61.252
deny from 72.49.0.0/16
deny from 76.27.180.212
deny from 74.54.133.98
deny from 74.217.98.0/24
deny from 74.226.28.93
deny from 72.70.183.39
deny from 72.91.234.78
deny from 72.205.253.44
deny from 72.224.0.0/13
deny from 72.226.191.224
deny from 72.232.212.210
deny from 72.232.233.118
deny from 72.249.60.74
deny from 74.52.0.0/14
deny from 74.53.244.18
deny from 74.54.99.194
deny from 74.125.184.83
deny from 75.98.9.254
deny from 75.101.128.0/17
deny from 75.101.197.161
deny from 75.101.187.208
deny from 75.73.0.0/16
deny from 76.2.14.128
deny from 76.18.2.60
deny from 76.110.63.107
deny from 77.0.0.0/8
deny from 77.224.224.206
deny from 78.137.163.133
deny from 79.0.0.0/8
deny from 80.0.0.0/8
deny from 80.40.134.120
deny from 81.0.0.0/8
deny from 81.30.64.0/20
deny from 81.52.143.15
deny from 81.52.143.16
deny from 81.104.144.187
deny from 81.226.182.173
deny from 82.11.64.47
deny from 84.10.217.140
deny from 85.0.0.0/8
deny from 85.17.216.105
deny from 86.0.0.0/8
deny from 86.129.34.170
deny from 86.147.183.9
deny from 87.0.0.0/8
deny from 87.90.45.43
deny from 87.118.108.229
deny from 88.198.25.25
deny from 88.198.43.39
deny from 88.247.50.225
deny from 88.255.173.104
deny from 89.0.0.0/8
deny from 91.0.0.0/8
deny from 91.205.96.19
deny from 91.205.124.15
deny from 92.0.0.0/8
deny from 92.61.149.46
deny from 92.112.0.0/15
deny from 93.0.0.0/8
deny from 93.189.56.0/21
deny from 95.0.0.0/8
deny from 101.0.0.0/8
deny from 107.20.0.0/14
deny from 107.170.0.0/1
deny from 107.170.0.0/16
deny from 109.0.0.0/8
deny from 109.228.93.239
deny from 113.212.64.0/19
deny from 113.212.68.243
deny from 117.0.0.0/8
deny from 119.0.0.0/8
deny from 119.63.196.8
deny from 122.0.0.0/8
deny from 123.0.0.0/8
deny from 128.121.191.197
deny from 128.208.3.93
deny from 129.128.25.237
deny from 129.242.108.35
deny from 130.0.0.0/16
deny from 132.178.248.47
deny from 134.245.38.100
deny from 136.169.0.0/16
deny from 139.149.31.232
deny from 141.0.0.0/8
deny from 142.245.0.0/16
deny from 142.104.35.118
deny from 140.109.19.172
deny from 142.91.0.0/16
deny from 142.91.79.216
deny from 142.206.0.0/16
deny from 144.76.13.77
deny from 146.102.32.72
deny from 150.0.0.0/8
deny from 150.1.0.0/16
deny from 150.70.172.204
deny from 151.237.189.212
deny from 155.69.13.195
deny from 159.226.0.0/16
deny from 162.83.223.242
deny from 162.216.96.0/21
deny from 165.21.0.0/16
deny from 169.203.0.0/16
deny from 170.140.76.176
deny from 171.0.0.0/8
deny from 173.192.0.0/15
deny from 173.192.111.19
deny from 173.199.64.0/18
deny from 173.208.96.0/24
deny from 173.208.97.0/24
deny from 173.234.14.0/23
deny from 173.234.178.0/24
deny from 173.248.128.0/18
deny from 173.252.64.0/18
deny from 174.129.0.0/16
deny from 174.133.5.250
deny from 174.133.177.66
deny from 176.0.0.0/8
deny from 176.9.192.110
deny from 180.0.0.0/8
deny from 182.0.0.0/8
deny from 183.0.0.0/8
deny from 184.72.0.0/15
deny from 184.173.232.61
deny from 184.173.249.171
deny from 185.0.0.0/8
deny from 188.0.0.0/8
deny from 189.31.113.110
deny from 189.153.116.115
deny from 190.16.41.251
deny from 190.156.58.141
deny from 190.156.59.103
deny from 190.223.133.206
deny from 192.31.21.179
deny from 192.85.50.2
deny from 192.96.200.0/21
deny from 192.99.19.154
deny from 192.113.0.0/16
deny from 192.114.0.0/15
deny from 192.116.0.0/15
deny from 192.118.0.0/16
deny from 192.187.122.126
deny from 192.199.240.0/20
deny from 193.47.80.0/24
deny from 193.252.149.15
deny from 193.252.149.16
deny from 194.0.0.0/8
deny from 194.27.13.195
deny from 194.67.32.120
deny from 194.153.113.18
deny from 194.187.183.244
deny from 194.213.194.207
deny from 196.0.0.0/8
deny from 196.207.248.205
deny from 196.0.0.0/8
deny from 198.54.0.0/16
deny from 198.143.130.125
deny from 198.61.224.117
deny from 198.168.48.35
deny from 199.19.72.0/21
deny from 199.19.74.220
deny from 199.19.73.12
deny from 199.19.248.0/21
deny from 199.46.128.0/17
deny from 199.46.198.230
deny from 199.46.199.230
deny from 199.112.24.47
deny from 199.192.72.0/24
deny from 199.213.0.0/16
deny from 199.214.0.0/15
deny from 199.216.0.0/16
deny from 199.233.239.128/25
deny from 200.0.0.0/8
deny from 201.2.235.88
deny from 201.34.147.142
deny from 201.2.235.88
deny from 201.88.0.130
deny from 201.215.128.73
deny from 201.233.94.203
deny from 201.240.42.242
deny from 202.0.0.0/7
deny from 204.236.128.0/17
deny from 204.248.120.202
deny from 204.8.168.0/22
deny from 204.8.217.126
deny from 204.15.20.0/22
deny from 204.174.223.201
deny from 204.187.154.49
deny from 204.248.120.202
deny from 205.236.3.0/24
deny from 205.236.3.72
deny from 206.83.86.2
deny from 206.196.111.0/24
deny from 207.46.13.68
deny from 208.17.184.0/24
deny from 208.50.101.152
deny from 208.66.193.175
deny from 208.80.192.0/21
deny from 208.90.56.0/21
deny from 208.91.113.20
deny from 208.96.0.0/18
deny from 208.96.10.200
deny from 208.96.54.70
deny from 208.96.54.71
deny from 208.101.28.131
deny from 208.115.96.0/19
deny from 208.115.111.0/24
deny from 208.115.111.240/28
deny from 209.62.0.0/17
deny from 209.62.68.194
deny from 209.133.64.214
deny from 209.137.230.23
deny from 209.216.154.110
deny from 209.249.86.17
deny from 209.249.86.210
deny from 210.82.118.14
deny from 210.205.6.206
deny from 211.239.124.204
deny from 212.52.148.115
deny from 212.113.32.0/20
deny from 213.0.0.0/8
deny from 216.152.240.0/20
deny from 217.0.0.0/8
deny from 217.67.229.133
deny from 217.77.220.156
deny from 218.0.0.0/8
deny from 218.111.64.230
deny from 218.85.139.170
deny from 218.186.10.11
deny from 219.0.0.0/8
deny from 220.0.0.0/8
deny from 220.129.0.0
deny from 220.181.19.172
deny from 221.2.243.114
deny from 222.124.198.0/24
deny from 222.153.166.29
allow from all
</Files>

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC]
RewriteCond %{HTTP_USER_AGENT} ^Charlotte/1.0b [OR]
RewriteCond %{HTTP_USER_AGENT} ^Charlotte [OR]
RewriteCond %{HTTP_USER_AGENT} ^jakarta [OR]
RewriteCond %{HTTP_USER_AGENT} ^Nutch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^talkdep.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^Gigamega [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^grub-client [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^grub-client [OR]
RewriteCond %{HTTP_USER_AGENT} ^grub-client-2.6.0 [OR]
RewriteCond %{HTTP_USER_AGENT} ^hget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^OmniExplorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^panscient.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^StackRambler [OR]
RewriteCond %{HTTP_USER_AGENT} ^asterias [OR]
RewriteCond %{HTTP_USER_AGENT} ^jakarta [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ripe.net/whois [OR]
RewriteCond %{HTTP_USER_AGENT} ^buytaert.net [OR]
RewriteCond %{HTTP_USER_AGENT} ^megabonk.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^pingdom.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^rr.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^test.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^apnic.net [OR]
RewriteCond %{HTTP_USER_AGENT} ^whois.sc [OR]
RewriteCond %{HTTP_USER_AGENT} ^rackforce.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^SBIder [OR]
RewriteCond %{HTTP_USER_AGENT} ^datacha0s [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^MSFrontPage [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*Indy [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^InternetSeer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^Ping [OR]
RewriteCond %{HTTP_USER_AGENT} ^Python-urllib/1.17 [OR]
RewriteCond %{HTTP_USER_AGENT} ^Link [OR]
RewriteCond %{HTTP_USER_AGENT} ^ia_archiver [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [OR]
RewriteCond %{HTTP_USER_AGENT} ^pycurl/ [NC]
RewriteCond %{HTTP_USER_AGENT} ^webcollage
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule !^http://[^/.].your-site.com.* - [F]

not2easy

2:55 am on Aug 14, 2014 (gmt 0)

I don't know where to start, I am surprised if you can see your own site with all of this. Your sever may just be giving up on processing all these rules in the time it is allowed to "decide" whether to let your friend in or not.

This is not a technologically complete or correct answer, but there are so many minor boo-boos in here it is hard to start with one.

But here goes - the order of your deny, allow is reversed. To block all those IPs you need to change:

order allow,deny

order deny,allow

It is very inefficient in such a long list of IPs and IP CIDRs that are being blocked, to feed them to the server out of sequential order. A few would probably not cause a problem, but there are thousands of IPs to check against and it has to be a problem to process all that when it needs to keep going back and checking numbers it has already checked. Here is one example of that:

deny from 204.15.80.0/22
deny from 58.0.0.0/8
deny from 198.103.53.0/24
deny from 67.202.0.0/18
deny from 72.44.32.0/19
deny from 79.125.0.0/17
deny from 174.129.0.0/16
deny from 204.236.128.0/17

After you have blocked every single IP address in the world that starts with 5. you don't need to block any more, but these are all in there:

deny from 5.0.0.0/8
deny from 5.135.125.179
deny from 5.135.125.182
deny from 5.135.125.183

You may have picked up bits and pieces from here, but this is your very own creation, probably unique in the world of htaccess files. I am surprised if it does not stop processing out of spite. It is very important and often repeated here to not ever paste things into your htaccess file without understanding two things:
What does this do? And is this something I need to do?

I'm not done, but giving you this information so you can begin mending your file. Until it is in working condition you might be better off to remove the .htaccess file because it definitely isn't doing what you wanted it to do. See if you can sort through this list and remove unnecessary and duplicate IPs for a start and remove that section for "User Agents" until we can get to it.

erlandc

3:12 am on Aug 14, 2014 (gmt 0)

gee thanks, ok, I've started putting them in sequential order & it'll take some time indeed. I should remove the "user agents" section? hmm, not sure exactly what you mean, but I'd probably mess it up. <sigh> thanks again!

not2easy

4:23 am on Aug 14, 2014 (gmt 0)

Yes, it is a lot of work, but it is important to have it done right or you really are better off without one.

The user agents section is the part below the allow,deny part with all the IPs.

From

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC]
RewriteCond %{HTTP_USER_AGENT} ^Charlotte/1.0b [OR]

RewriteCond %{HTTP_USER_AGENT} ^webcollage
RewriteRule ^.* - [F]

This is a very inefficient way to block USER_AGENTs (which means browsers) you want to block. There are domains listed there that will never be found in a USER_AGENT and even evil agents that don't start their names that way so they get a pass.

at the start of the name you have an anchor: ^ which means that the server should only block that browser if the UA begins with the name you show. If it starts off with anything except that name, it gets in. The better way to do that is to make your own list from checking your access logs to see what UAs are up to no good on your site. Then you put them in a list that only needs to contain (not start with) that UA name. That would cut your list down to a more manageable size. Instead of:

RewriteCond %{HTTP_USER_AGENT} ^grub-client [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^grub-client [OR]
RewriteCond %{HTTP_USER_AGENT} ^grub-client-2.6.0 [OR]

(which I've been lucky enough to have never seen) you would only need to use

|grub|

to block them when used in a list. Think about making a few rows like this one:

RewriteCond %{HTTP_USER_AGENT} (fetch|filter|flip|genieo|grub) [NC,OR]

instead of checking them one at a time, the server can check against more than one and move to the next batch. The pipe character "|" means (or) and [NC, at the end means "No Case" so Fetch or fetch or fEtch found within a UA will be blocked. The OR] part means "or".

When you get to the end of your list of UAs, this gives them the 403:

RewriteRule .* - [F]

You did have the flag correct in that the last one in the list does not need the OR] part.

I apologize, but I don't have time today to dissect this rule, but it is not doing what it looks like you want it to do:

RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule !^http://[^/.].your-site.com.* - [F]

When you get the list tamed you should look into adding the basic canonicalization rules so you won't have 4 home pages. If this really is the whole htaccess file you can access your site by www.example.com or example.com or www.example.com/index.(html|php) or or example.com/index.(html|php) which can be confusing and can make people unsure of what to link to if they want to link to your site.

erlandc

4:40 am on Aug 14, 2014 (gmt 0)

WOW! Looks like I got my work cut out for me! Thanks alot! I'll try to decipher what you've written & fix what I've messed up. I feel like Gomer Pyle: "gawlee"!

Wish I could talk the way you experts talk - let alone get it in my head - like JDMorgan & others who have helped me before on this stuff. (I'm rolling up my sleeves!) Thanks so much not2easy!

not2easy

5:35 am on Aug 14, 2014 (gmt 0)

Gee whiz... I'm no expert, but I read and learn here all the time. Eventually it sinks in and makes sense. When you've whittled it down a little, drop back and we can help you wrap it up. Most of everything I shared with you was shared with me by one of the experts here. It is all spelled out clearly sooner or later.

erlandc

5:46 am on Aug 14, 2014 (gmt 0)

Ok gotcha! I try to learn here too, bit the experts are waaaaaaay ahead of me, so thanks to them as well for helping you help me!
They know who they are! lucy is one methinks. I'll be lining up those IPs in order! :(

penders

11:05 am on Aug 14, 2014 (gmt 0)

But here goes - the order of your deny, allow is reversed. To block all those IPs you need to change:
order allow,deny to
order deny,allow

Although wouldn't this alone result in everything being allowed ...since there is an

allow from all

directive at the bottom? I would assume that if you change the

order

directive, then the

allow from all

directive should also be removed? (So, shouldn't it be working in its current state - but logic is just reversed?)

...it is not doing what it looks like you want it to do:
RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule !^http://[^/.].your-site.com.* - [F]

Yes, that RewriteRule pattern would seem to be entirely superfluous / misleading. This looks like it will simply block all traffic from

http://www.iaea.org

, the RewriteRule pattern will always match (yes, the URL to be matched is never "http://www.your-site.com" - the periods are not escaped either). As far as I can tell, this is equivalent to:

RewriteCond %{HTTP_REFERER} ^http://www.iaea.org$
RewriteRule .* - [F]

wilderness

12:15 pm on Aug 14, 2014 (gmt 0)

I've started putting them in sequential order & it'll take some time indeed.

FWIW, you may use the sort function in Word, however when completed MAKE SURE that you copy the Word sort back into a new Notepad file (txt), and then from that new NotePad (text)copy the lines into your htaccess.

wilderness

12:20 pm on Aug 14, 2014 (gmt 0)

^http://www.iaea.org$

I've not seen this in some while.
Years ago somebody use to use this to antagonize people or make some obscure political gesture.

not2easy

12:34 pm on Aug 14, 2014 (gmt 0)

You are right there, penders- because my own htaccess uses SetEnvIf before that and ends with Allow from env= rather than Allow from all it is not the same as using the mod_authz_host only. Thanks for that check. It would make a difference.

erlandc

5:28 pm on Aug 14, 2014 (gmt 0)

Thanks penders: I'll try to figure out what you wrote as I'm no pro at this stuff. (I'm always willing & eager to learn! from you all)

wilderness: I got that "^http://www.iaea.org$" from an .htaccess file from webmasterworld a few years ago & modified it, obviously wrongly. Thanks for your input & will again, learn. WHOOPS! Before I went to bed, I did think of sorting the IPs in Excel, but now I'll use WORD. Thanks!

not2easy: thanks again for your help!

Now, I'm back to the grindstone...

erlandc

7:34 pm on Aug 14, 2014 (gmt 0)

Hi all,
Unable to sort in WORD or EXCEL, so will do it manually. Talk to you in about a week. (lol!) Thanks again!

not2easy

8:02 pm on Aug 14, 2014 (gmt 0)

OK, here it is, I wouldn't want you to spend a week without any htaccess file - I use BBEdit on Mac and it sorts pretty quickly.
deny from 1.0.0.0/8
deny from 5.0.0.0/8
deny from 5.135.125.179
deny from 5.135.125.182
deny from 5.135.125.183
deny from 8.18.144.0/23
deny from 23.20.0.0/14
deny from 23.20.0.0/14
deny from 23.22.163.55
deny from 24.4.111.95
deny from 24.58.0.0/15
deny from 24.59.94.137
deny from 24.125.192.215
deny from 24.125.202.29
deny from 24.136.95.151
deny from 24.168.35.217
deny from 24.232.0.0/16
deny from 31.0.0.0/8
deny from 37.0.0.0/8
deny from 38.0.0.0/8
deny from 38.113.234.181
deny from 41.0.0.0/8
deny from 46.51.128.0/18
deny from 46.51.192.0/20
deny from 46.51.208.0/22
deny from 46.51.212.0/23
deny from 46.51.214.0/24
deny from 46.51.215.0/25
deny from 46.51.215.128/26
deny from 46.51.215.192/27
deny from 46.51.215.224/28
deny from 46.51.215.240/29
deny from 46.51.215.248/30
deny from 46.51.215.252/31
deny from 46.51.215.254/32
deny from 46.51.216.0/21
deny from 46.51.224.0/20
deny from 46.137.0.0/16
deny from 54.72.0.0/13
deny from 54.80.0.0/12
deny from 54.80.214.201
deny from 54.176.0.0/12
deny from 54.204.0.0/15
deny from 54.221.0.0/16
deny from 54.224.0.0/11
deny from 54.224.0.0/12
deny from 54.234.0.0/15
deny from 54.240.0.0/12
deny from 58.0.0.0/8
deny from 59.95.210.66
deny from 60.0.0.0/8
deny from 60.48.69.4
deny from 60.50.61.192
deny from 60.191.80.45
deny from 60.240.248.19
deny from 60.240.248.23
deny from 60.240.248.24
deny from 61.247.219.44
deny from 61.247.219.50
deny from 63.92.12.0/22
deny from 63.209.222.18
deny from 63.238.12.0/22
deny from 63.238.16.0/23
deny from 63.251.92.200
deny from 64.111.127.81
deny from 64.124.0.0/15
deny from 64.125.222.16
deny from 64.211.0.0/17
deny from 64.211.112.0/21
deny from 64.211.128.0/18
deny from 64.211.192.0/19
deny from 64.213.218.112
deny from 64.229.86.42
deny from 64.233.160.0/19
deny from 64.235.144.0/20
deny from 65.28.108.199
deny from 65.52.0.0/14
deny from 65.78.161.42
deny from 65.92.28.37
deny from 65.92.216.70
deny from 65.98.224.3
deny from 65.210.123.237
deny from 65.214.39.180
deny from 66.7.64.0/19
deny from 66.34.204.26
deny from 66.45.252.90
deny from 66.90.118.101
deny from 66.96.128.0/18
deny from 66.111.32.0/19
deny from 66.120.0.0/13
deny from 66.130.218.119
deny from 66.136.0.0/13
deny from 66.147.244.213
deny from 66.150.0.0/15
deny from 66.150.224.245
deny from 66.175.164.24
deny from 66.182.88.94
deny from 66.201.230.0/24
deny from 66.212.253.254
deny from 66.212.254.217
deny from 66.220.144.0/20
deny from 66.228.202.22
deny from 66.228.210.10
deny from 66.228.210.50
deny from 66.232.111.94
deny from 67.15.0.0/16
deny from 67.55.72.165
deny from 67.70.153.23
deny from 67.78.34.170
deny from 67.86.56.104
deny from 67.168.154.188
deny from 67.202.0.0/18
deny from 67.202.0.0/18
deny from 67.202.0.0/18
deny from 67.202.0.0/18
deny from 67.202.24.96
deny from 67.202.25.192
deny from 67.202.49.33
deny from 67.202.56.158
deny from 67.205.69.32/27
deny from 67.205.96.152
deny from 67.215.234.250
deny from 67.228.100.141
deny from 69.9.181.169
deny from 69.11.119.107
deny from 69.25.142.42
deny from 69.36.144.0/20
deny from 69.36.158.36
deny from 69.37.0.0/16
deny from 69.37.32.0/23
deny from 69.58.176.0/20
deny from 69.58.178.29
deny from 69.63.176.0/20
deny from 69.64.145.225
deny from 69.90.103.173
deny from 69.149.107.70
deny from 69.162.128.0/18
deny from 69.171.224.0/19
deny from 69.211.0.0/20
deny from 70.29.145.119
deny from 70.29.148.187
deny from 70.33.160.0/19
deny from 70.33.184.32/27
deny from 70.42.37.7
deny from 70.42.186.129
deny from 70.48.219.155
deny from 70.84.0.0/14
deny from 70.85.81.76
deny from 70.87.15.50
deny from 70.113.80.43
deny from 70.168.200.180
deny from 70.176.70.77
deny from 70.240.0.0/12
deny from 71.13.115.117
deny from 71.71.7.51
deny from 71.197.96.124
deny from 71.202.29.212
deny from 71.204.21.71
deny from 71.232.218.119
deny from 72.8.151.0/24
deny from 72.8.191.0/24
deny from 72.13.32.0/19
deny from 72.21.192.0/19
deny from 72.29.185.0/24
deny from 72.36.64.0/18
deny from 72.36.180.234
deny from 72.44.32.0/19
deny from 72.44.32.0/19
deny from 72.44.32.0/19
deny from 72.44.58.248
deny from 72.44.61.252
deny from 72.49.0.0/16
deny from 72.55.128.0/18
deny from 72.70.183.39
deny from 72.91.234.78
deny from 72.205.253.44
deny from 72.224.0.0/13
deny from 72.226.191.224
deny from 72.232.212.210
deny from 72.232.233.118
deny from 72.249.60.74
deny from 74.52.0.0/14
deny from 74.53.244.18
deny from 74.54.99.194
deny from 74.54.133.98
deny from 74.125.184.83
deny from 74.217.98.0/24
deny from 74.226.28.93
deny from 75.73.0.0/16
deny from 75.98.9.254
deny from 75.101.128.0/17
deny from 75.101.187.208
deny from 75.101.197.161
deny from 76.2.14.128
deny from 76.18.2.60
deny from 76.27.180.212
deny from 76.110.63.107
deny from 77.0.0.0/8
deny from 77.224.224.206
deny from 78.137.163.133
deny from 79.0.0.0/8
deny from 79.125.0.0/16
deny from 79.125.0.0/17
deny from 80.0.0.0/8
deny from 80.40.134.120
deny from 81.0.0.0/8
deny from 81.30.64.0/20
deny from 81.52.143.15
deny from 81.52.143.16
deny from 81.104.144.187
deny from 81.144.138.34
deny from 81.144.138.40
deny from 81.226.182.173
deny from 82.11.64.47
deny from 84.10.217.140
deny from 85.0.0.0/8
deny from 85.17.216.105
deny from 86.0.0.0/8
deny from 86.129.34.170
deny from 86.147.183.9
deny from 87.0.0.0/8
deny from 87.90.45.43
deny from 87.118.108.229
deny from 87.231.235.2/32
deny from 88.198.25.25
deny from 88.198.43.39
deny from 88.247.50.225
deny from 88.255.173.104
deny from 89.0.0.0/8
deny from 91.0.0.0/8
deny from 91.205.96.19
deny from 91.205.124.15
deny from 92.0.0.0/8
deny from 92.61.149.46
deny from 92.112.0.0/15
deny from 93.0.0.0/8
deny from 93.189.56.0/21
deny from 95.0.0.0/8
deny from 101.0.0.0/8
deny from 107.20.0.0/14
deny from 107.20.0.0/14
deny from 107.20.0.0/14
deny from 107.170.0.0/1
deny from 107.170.0.0/16
deny from 109.0.0.0/8
deny from 109.228.93.239
deny from 113.212.64.0/19
deny from 113.212.68.243
deny from 117.0.0.0/8
deny from 119.0.0.0/8
deny from 119.63.196.8
deny from 122.0.0.0/8
deny from 123.0.0.0/8
deny from 128.121.191.197
deny from 128.208.3.93
deny from 129.128.25.237
deny from 129.242.108.35
deny from 130.0.0.0/16
deny from 132.178.248.47
deny from 134.245.38.100
deny from 136.169.0.0/16
deny from 139.149.31.232
deny from 140.109.19.172
deny from 141.0.0.0/8
deny from 142.91.0.0/16
deny from 142.91.79.216
deny from 142.104.35.118
deny from 142.206.0.0/16
deny from 142.245.0.0/16
deny from 144.76.13.77
deny from 146.102.32.72
deny from 150.0.0.0/8
deny from 150.1.0.0/16
deny from 150.70.172.204
deny from 151.237.189.212
deny from 155.69.13.195
deny from 159.226.0.0/16
deny from 162.83.223.242
deny from 162.216.96.0/21
deny from 164.138.209.245
deny from 165.21.0.0/16
deny from 169.203.0.0/16
deny from 170.140.76.176
deny from 171.0.0.0/8
deny from 173.192.0.0/15
deny from 173.192.111.19
deny from 173.199.64.0/18
deny from 173.208.96.0/24
deny from 173.208.97.0/24
deny from 173.234.14.0/23
deny from 173.234.178.0/24
deny from 173.248.128.0/18
deny from 173.252.64.0/18
deny from 174.129.0.0/16
deny from 174.129.0.0/16
deny from 174.129.0.0/16
deny from 174.133.5.250
deny from 174.133.177.66
deny from 176.0.0.0/8
deny from 176.9.192.110
deny from 180.0.0.0/8
deny from 182.0.0.0/8
deny from 183.0.0.0/8
deny from 184.72.0.0/15
deny from 184.72.0.0/15
deny from 184.72.0.0/15
deny from 184.173.232.61
deny from 184.173.249.171
deny from 185.0.0.0/8
deny from 188.0.0.0/8
deny from 189.31.113.110
deny from 189.153.116.115
deny from 190.16.41.251
deny from 190.156.58.141
deny from 190.156.59.103
deny from 190.223.133.206
deny from 192.31.21.179
deny from 192.85.50.2
deny from 192.96.200.0/21
deny from 192.99.19.154
deny from 192.113.0.0/16
deny from 192.114.0.0/15
deny from 192.116.0.0/15
deny from 192.118.0.0/16
deny from 192.187.122.126
deny from 192.199.240.0/20
deny from 193.47.80.0/24
deny from 193.252.149.15
deny from 193.252.149.16
deny from 194.0.0.0/8
deny from 194.27.13.195
deny from 194.67.32.120
deny from 194.153.113.18
deny from 194.187.183.244
deny from 194.213.194.207
deny from 196.0.0.0/8
deny from 196.0.0.0/8
deny from 196.207.248.205
deny from 198.54.0.0/16
deny from 198.61.224.117
deny from 198.103.53.0/24
deny from 198.143.130.125
deny from 198.168.48.35
deny from 199.19.72.0/21
deny from 199.19.73.12
deny from 199.19.74.220
deny from 199.19.248.0/21
deny from 199.46.128.0/17
deny from 199.46.198.230
deny from 199.46.199.230
deny from 199.112.24.47
deny from 199.192.72.0/24
deny from 199.213.0.0/16
deny from 199.214.0.0/15
deny from 199.216.0.0/16
deny from 199.233.239.128/25
deny from 200.0.0.0/8
deny from 201.2.235.88
deny from 201.2.235.88
deny from 201.34.147.142
deny from 201.88.0.130
deny from 201.215.128.73
deny from 201.233.94.203
deny from 201.240.42.242
deny from 202.0.0.0/7
deny from 204.8.168.0/22
deny from 204.8.217.126
deny from 204.15.20.0/22
deny from 204.15.80.0/22
deny from 204.15.80.50
deny from 204.74.108.0/24
deny from 204.174.223.201
deny from 204.187.154.49
deny from 204.236.128.0/17
deny from 204.236.128.0/17
deny from 204.236.128.0/17
deny from 204.236.128.0/17
deny from 204.236.224.0/19
deny from 204.246.160.0/22
deny from 204.246.167.0/24
deny from 204.246.168.0/23
deny from 204.246.176.0/21
deny from 204.246.184.0/22
deny from 204.248.120.202
deny from 204.248.120.202
deny from 205.236.3.0/24
deny from 205.236.3.72
deny from 206.83.86.2
deny from 206.196.111.0/24
deny from 207.46.13.68
deny from 207.171.160.0/19
deny from 208.17.184.0/24
deny from 208.47.248.0/23
deny from 208.50.101.152
deny from 208.66.193.175
deny from 208.80.192.0/21
deny from 208.90.56.0/21
deny from 208.91.113.20
deny from 208.96.0.0/18
deny from 208.96.10.200
deny from 208.96.54.70
deny from 208.96.54.71
deny from 208.101.28.131
deny from 208.115.96.0/19
deny from 208.115.111.0/24
deny from 208.115.111.240/28
deny from 209.62.0.0/17
deny from 209.62.68.194
deny from 209.133.64.214
deny from 209.137.230.23
deny from 209.201.96.0/22
deny from 209.216.154.110
deny from 209.249.86.17
deny from 209.249.86.210
deny from 210.82.118.14
deny from 210.205.6.206
deny from 211.239.124.204
deny from 212.52.148.115
deny from 212.113.32.0/20
deny from 213.0.0.0/8
deny from 216.137.32.0/20
deny from 216.137.48.0/21
deny from 216.152.240.0/20
deny from 216.182.224.0/20
deny from 217.0.0.0/8
deny from 217.67.229.133
deny from 217.77.220.156
deny from 218.0.0.0/8
deny from 218.85.139.170
deny from 218.111.64.230
deny from 218.186.10.11
deny from 219.0.0.0/8
deny from 220.0.0.0/8
deny from 220.129.0.0
deny from 220.181.19.172
deny from 221.2.243.114
deny from 222.124.198.0/24
deny from 222.153.166.29

As you're going through your list, remember that 0/8 at the end means you have blocked all of them so the next row starts with some other number. That is why after you use:

deny from 217.0.0.0/8

You don't need:

deny from 217.67.229.133
deny from 217.77.220.156

erlandc

8:36 pm on Aug 14, 2014 (gmt 0)

WHEW! thanks again not2easy! I friend of mine told me to get a mac, word & excel was a hassle, maybe 'cos it couldn't sort because of the dots & slashes. Going to upload now & get to that other lingo you, wilderness & penders wrote. cheers!

not2easy

8:55 pm on Aug 14, 2014 (gmt 0)

I would not upload anything until you pass it through here again. I sorted your list, that's all, you need to go through the list and remove all those duplicates and make sure that is all you wanted to block. The list I gave you here is definitely not ready to upload, I just helped you get started.

IF you didn't get those ips and CIDRs from your own logs and lookups you may be wasting your servers hard work for nothing while the ones you really don't want inside are going right in.

Honest, we aren't nosey, just don't want things to go wrong if we can help you get it right.

erlandc

9:12 pm on Aug 14, 2014 (gmt 0)

ok thanks! I haven't uploaded it yet - and yes, the IPs & CIDRs are from my own logs. No need to feel nosey, I'm ok with all ya'll say here.

I'm still puzzled as to why my friend can't access my site. I know his IP obviously & it's not in my htaccess file, so I'll surmise it's in the "RewriteEngine on" file. My head hurts now. Thanks again!

lucy24

10:20 pm on Aug 14, 2014 (gmt 0)

(\d+)\.0\.0\.0/8
>>
$1
e.g.
Deny from 5.0.0.0/8
=
Deny from 5

and similarly for
(\d+\.\d+)\.0\.0/16
and
(\d+\.\d+\.\d+)\.0/24

The list of IPs is much, much too long. See the ongoing Server Farms thread in the Search Engine Spiders subforum for whole ranges to block. I got as far as

Deny from 1
Deny from 5
Deny from 5.135.125
Deny from 8.18.144.0/23
Deny from 23.20.0.0/14

Deny from 24.4.111.95
Deny from 24.58.0.0/15
Deny from 24.125.192.0/20
Deny from 24.136.95.151
Deny from 24.168.35.217
Deny from 24.232
Deny from 31
Deny from 37
Deny from 38
Deny from 38.113.234.181
Deny from 41.0.0.0/8
Deny from 46.51.128.0/17
Deny from 46.137.0.0/16
Deny from 54.72.0.0/13
Deny from 54.80.0.0/12
Deny from 54.176.0.0/12
Deny from 54.204.0.0/15
Deny from 54.221.0.0/16
Deny from 54.224.0.0/11

... and then I got tired. Some bits were admittedly guessing: if you're blocking a, b, c and e, there's probably no reason to admit d. That kind of thing.

And don't even bother with exact 123.123.123.123 IPs unless there's some absolutely extraordinary circumstance. Find the sector and block it. Otherwise you're looking at
2^32
possibilities before you even get to the IPv6 lockouts.

Any self-respecting text editor should have a Sort function somewhere. The trick with CIDRs is that you may need to pad out ranges, for example .2. >> .002. and .12. >> .012. so things get sorted correctly. (Yes, of course I have a Regular Expression that does this ;))

I know his IP obviously & it's not in my htaccess file, so I'll surmise it's in the "RewriteEngine on" file. My head hurts now.

If you're restricted to htaccess and logs, you're pretty well stuck. Logs will never say more than "Client denied by server configuration", thank you very much logs, that much I'd figured out already. A RewriteLog (config file only) could give you more information.

Is there anything unusual in your friend's UA string, such as an addon or plugin that happens to share a name with some undesirable robot? Have they tried a different browser?

Further Edit:
This part is personal preference, so expect so see some squabbling between me & Wilderness on this point, but I consider mod_rewrite a last resort when it comes to access control. Most simple UA-based blocks can be done just as easily with a combination of mod_setenvif and mod_authz-thingummy.

BrowserMatch libwww-perl keep_out
BrowserMatch "MSIE [1-6]\." keep_out
etcetera through all the basics, and then

Deny from env=keep_out

where "keep_out" is any name you choose to use for your environmental variable. Another popular one is "bad_bot". Note that you don't have to give it any particular value; it's enough to say "bring it into existence".

If you have lines that say
Deny from blahblah
then the ordering has to be
Order Allow,Deny
with an
Allow from all
somewhere in the mix. The opposite approach, whitelisting, is

Order Deny,Allow
Deny from all
Allow from blahblah

erlandc

11:35 pm on Aug 14, 2014 (gmt 0)

Hi lucy, thanks for your input time before you got tired. sorry 'bout that. again, I'll have to decipher the lingo & figure it out. sorry I'm such a dough-head with htaccess stuff. :(

lucy24

12:33 am on Aug 15, 2014 (gmt 0)

Clearly I got tired even sooner than I thought :(

Deny from 1
Deny from 5
Deny from 5.135.125
Deny from 8.18.144.0/23
Deny from 23.20.0.0/14

Deny from 24.4.111.95
Deny from 24.58.0.0/15
Deny from 24.125.192.0/20
Deny from 24.136.95.151
Deny from 24.168.35.217
Deny from 24.232
Deny from 31
Deny from 37
Deny from 38
Deny from 38.113.234.181

erlandc

12:39 am on Aug 15, 2014 (gmt 0)

hehe, well, now what you have in that box is a wonder to me. Wish I could speak your language. :( parle vous francais?

not2easy

1:49 am on Aug 15, 2014 (gmt 0)

Translation:
Deny from 5.0.0.0/8 = Deny from 5
You can trim that Deny from 5.0.0.0/8 to Deny from 5 and have the same results.

An idea (I have not tried this, but it might be a quick answer for you.) You use Windows, so download PSPad which is a free text editor I have used on Windows. Load a copy of those lines into PSPad and use the menu to "Find and Replace using Regex" and try the regex formulas Lucy posted above to see if that takes care of the list for you. Time to learn new stuff? It can help in a lot of other things once you get familiar with it.

erlandc

2:40 am on Aug 15, 2014 (gmt 0)

ok thanks not2easy for that & that PSPad idea. More stuff to learn! yikes! My head is splitting! LOL! you rock! et al!