'including' blocklist in httpd.conf - am I doing it correctly?

[Mark_Young]

I have a dedicated server (CentOS/WHM/cPanel) I use to host a handful of websites.

Being fairly new to hosting, and after noticing I was getting a lot of unnecessary views/clicks from a lot of similar IP's and user-agents I decided to look in to what I could do to stop them.

I initially decided to go with this: [github.com...] which seemed to be the answers to my IP block problem. Every time I found another offending IP I would check it's source and, if I thought it needed it, add it's whole block in CIDR format to the blacklist.

I had been doing this for a number of weeks when I then wanted to tackle the user-agent problem. Again I did some reading up and put together a solution I thought was working.

I carried on with this for a number of months, but just lately I have noticed I am receiving clicks from IP's that should be blocked. I've tried all I can to work out why this is happening, but cannot find a solution.

My search to find an answer to my problem has lead me to find out there are many lists available that would help me, such as Hostile Servers Blocklists (Listing IP ranges of Dedicated Server & VPS Server hosting companies who have been, and are repeatedly used for various attacks and abuse), Bad Bots + Parasites Blocklists etc.

This has lead me to question how I am going about things, as firstly the blocks I am adding don't always seem to work, and secondly whether adding lots of IP's to the list is going to cause slowdown to he server - Maybe there is a better way to go about the whole allow/deny setup I am currently running?

Can anybody who knows cast their eye over what I have done so far and let me know if what I am doing is correct please (and if so, why am I still seeing clicks from blocked IP's)?

Also, as I'm thinking of adding a lot more IP's into the block list, is there a better way for me to achieve what I am trying to do?

Many thanks for any help with this.

*I will attach the files in separate posts as it's making this one too big!

[Mark_Young]

blacklist.conf

#
# This file is comprised of blacklisted IPs based on honeypot site visits by various bots.
#
#
# INSTALLATION INSTRUCTIONS
#
# To use it, copy it to your Apache http's conf.d dir:
# <ServerRoot>/conf.d/ip_blacklist.conf
#
# Ensure that it is included from the httpd.conf file. Look for the following line:
# Include conf.d/*.conf
#
# If it isn't present, you can explicitly include this file with the following line:
# Include conf.d/ip_blacklist.conf
#
# This file applies the blacklisted IPs to every single server request, regardless of what
# location the user agent requests. It will also override the access rules defined in
# directives such as <Directory> and <Files>. The following Apache documentation is helpful
# to understand how these directives are merged and override each other.
# http://httpd.apache.org/docs/2.2/sections.html#mergin
#
# You might also include this file from an .htaccess file, but this is not quite as
# efficient as having httpd load the configuration on startup because it will read the
# .htaccess file every time a request is made to the server.
#
#
# LICENSE AND COPYRIGHT
#
# Copyright (c) 2012 Pushing Inertia
# http://pushinginertia.com
#
# Released under the MIT license.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of this
# software and associated documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons
# to whom the Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all copies or
# substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
# PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
# FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
#
# The latest version of this file is available on GitHub:
# https://github.com/pushinginertia/ip-blacklist
#

<Location />

order allow,deny

# fibercloud
deny from 216.145.0.0/19

# Verisign
deny from 69.58.176.0/20

# corexchange.com
deny from 198.154.96.0/19

# multacom.com
deny from 204.13.152.0/22
deny from 198.52.100.0/26

# Name Intelligence (domaintools) 
deny from 64.246.165.128/25

# micfo.com
deny from 192.240.208.0/21
deny from 23.232.148.0/22
deny from 192.240.192.0/18

# amazonaws
deny from 54.80.0.0/12

# QuadraNet.com
deny from 198.55.96.0/19
deny from 155.94.128.0/17

# continuumdatacenters.com
deny from 216.107.144.0/20

# techiemedia.net
deny from 199.80.56.0/21

# sprintdatacenter.pl customer
deny from 185.38.251.0/24

# e-ring.pl
deny from 91.188.117.0/24

# quadranet
deny from 72.11.128.0/19

# turnkey internet
deny from 208.85.0.0/21

# www.cscprotectsbrands.com
deny from 99.95.152.169

# AD Technology
deny from 188.92.76.0/24

# poweruphosting.com
deny from 162.244.8.0/21

# PSINet, Inc.
deny from 38.100.21.0/24

# fastweb.it (part)
deny from 2.229.50.16/30

# portlane.com (PRIVACTUALLY-NET)
deny from 46.246.32.0/19 

# fdcservers.net
deny from 50.7.0.0/16

# supremebytes
deny from 23.92.48.0/20 

# wowrack.com
deny from 216.244.64.0/19 

# PacketExchange
deny from 68.64.128.0/18 

# Enzu
deny from 23.88.0.0/15

# IPStrada
deny from 199.15.232.0/21 

# relakks.com
deny from 93.0.0.0/8

# erank? (polish)
deny from 91.236.74.0/23

# leasweb (one IP)
deny from 207.244.73.5

# Xeex (one IP)
deny from 216.152.251.91

# Quadranet
deny from 69.12.64.0/19

# poweruphosting
deny from 162.244.8.0/21

# idealhosting.net.tr (buyURL)
deny from 213.238.175.0/24

# versaweb
deny from 76.164.224.0/20
deny from 76.164.192.0/19

# Google Cloud
deny from 23.251.128.0/19

# Digital Ocean
deny from 198.211.96.0/19
deny from 198.199.64.0/18
deny from 162.243.0.0/16
deny from 192.34.56.0/21

# datashack.net
deny from 107.150.32.0/19
deny from 192.187.96.0/19

# Part of Hetzner
deny from 188.40.0.0/16

# Zhou Pizhong Datashack
deny from 192.187.120.64/29

# Phychz Networks
deny from 23.238.128.0/17

# Linode
deny from 96.126.96.0/19
deny from 23.239.0.0/19

# OVH Hosting
deny from 198.27.64.0/18
deny from 198.50.128.0/17
deny from 198.50.175.0/24
deny from 198.100.144.0/20
deny from 198.245.48.0/20
deny from 192.99.0.0/16

# PhishMongers
deny from 198.186.194.0/24
deny from 198.186.190.0/23
deny from 198.186.192.0/23

# AWS: http://www.forumpostersunion.com/showthread.php?t=10490
deny from 23.20.0.0/14
deny from 46.51.128.0/17
deny from 46.137.0.0/16
deny from 50.16.0.0/14
deny from 50.19.0.0/16
deny from 54.224.0.0/12
deny from 54.240.0.0/12
deny from 67.202.0.0/18
deny from 72.44.32.0/19
deny from 75.101.128.0/17
deny from 79.125.0.0/17
deny from 103.4.8.0/21
deny from 107.20.0.0/14
deny from 122.248.192.0/18
deny from 174.129.0.0/16
deny from 175.41.128.0/17
deny from 176.32.64.0/18
deny from 176.34.0.0/16
deny from 184.72.0.0/15
deny from 204.236.128.0/17
deny from 216.182.224.0/20

# Rackspace Cloud Servers: http://www.encoding.com/what_is_the_ip_range_of_your_system_so_i_may_restrict_access_on_a_firewall
deny from 50.56.0.0/15
deny from 173.203.194.0/19
deny from 174.143.0.0/16
deny from 184.106.0.0/16
deny from 204.232.192.0/18
deny from 209.114.32.0/19

# reliablehosting
deny from 66.151.32.0/22
deny from 68.68.32.0/20
deny from 98.158.112.0/20
deny from 207.204.224.0/19
deny from 216.131.64.0/18

# AS47142 steephost.com
deny from 91.207.4.0/22
deny from 91.207.8.0/23
deny from 195.190.13.0/24

# egihosting.com AS18779
deny from 50.117.0.0/17
deny from 64.145.82.0/23
deny from 68.68.96.0/20
deny from 69.46.64.0/19
deny from 173.245.64.0/19
deny from 205.164.0.0/18
deny from 209.73.128.0/19
deny from 216.151.191.0/24
deny from 216.172.128.0/19

# datacom bulgaria
deny from 213.91.181.0/24

# snelserver.com
deny from 77.95.224.0/21
deny from 78.41.200.0/21
deny from 89.207.128.0/21
deny from 128.204.192.0/20
deny from 178.255.198.0/23
deny from 193.34.166.0/23

# everhost.ro
deny from 94.60.152.0/21
deny from 94.60.160.0/19
deny from 94.60.192.0/21
deny from 94.63.0.0/19
deny from 94.63.32.0/20
deny from 94.63.56.0/21
deny from 94.63.64.0/21
deny from 94.177.4.0/23
deny from 188.215.32.0/22

# AS35662 redstation.co.uk
deny from 31.3.224.0/19
deny from 80.84.48.0/20
deny from 80.243.176.0/20
deny from 109.73.64.0/20
deny from 149.3.128.0/20
deny from 188.227.160.0/19

# softlayer
deny from 50.22.0.0/15
deny from 50.97.0.0/16
deny from 67.228.0.0/16
deny from 75.126.0.0/16
deny from 159.253.0.0/16
deny from 173.192.0.0/15
deny from 174.36.0.0/15
deny from 174.127.64.0/18
deny from 184.172.0.0/15
deny from 198.58.80.0/20
deny from 208.43.0.0/16
deny from 208.101.0.0/18

# nexcess.net
deny from 69.160.48.0/20

# ubiquity / nobis technology group AS15003
deny from 23.19.0.0/16
deny from 64.120.0.0/17
deny from 69.147.224.0/19
deny from 108.62.0.0/16
deny from 173.208.0.0/17
deny from 173.234.0.0/16
deny from 174.34.128.0/18
deny from 216.6.224.0/20

# AS8551 bzqint-dc
deny from 62.219.0.0/20
deny from 62.219.16.0/22
deny from 62.219.20.0/23
deny from 79.176.0.0/15
deny from 79.179.0.0/16
deny from 79.180.0.0/14
deny from 81.218.192.0/17
deny from 82.80.0.0/15
deny from 84.108.0.0/14
deny from 85.130.128.0/17
deny from 109.64.0.0/15
deny from 192.114.64.0/20

# AS6724 STRATO-RZG-MNT http://www.strato.de/
deny from 81.169.128.0/17
deny from 85.214.0.0/16

# NTT PC Communications,Inc. www.nttpc.co.jp/english/
deny from 119.245.0.0/16

# AS49335 nconnect.ru/hostkey.com
deny from 146.0.72.0/22
deny from 31.192.104.0/21
deny from 46.17.96.0/21
deny from 91.210.104.0/22
deny from 141.105.64.0/21
deny from 158.255.0.0/21
deny from 195.162.68.0/23

# AS30633 leaseweb.com (previously netdirekt)
deny from 46.165.192.0/18
deny from 62.212.64.0/19
deny from 78.159.96.0/19
deny from 83.149.64.0/18
deny from 85.17.0.0/16
deny from 89.149.192.0/18
deny from 91.109.16.0/20
deny from 95.168.166.0/23
deny from 108.59.0.0/20
deny from 130.185.76.0/23
deny from 162.210.192.0/21
deny from 178.162.128.0/17
deny from 185.17.144.0/22
deny from 192.96.200.0/21
deny from 198.7.56.0/21
deny from 199.58.84.0/22
deny from 199.115.112.0/21
deny from 207.244.64.0/18
deny from 209.58.128.0/18

# AS23033 Wowrack.com WOW-IPV4-NET2
deny from 208.115.96.0/19

# AS40676 psychz.net
deny from 216.24.192.0/20
deny from 216.24.240.0/20
deny from 208.87.240.0/22
deny from 74.117.56.0/21
deny from 173.224.208.0/20
deny from 199.15.112.0/21
deny from 199.83.88.0/21
deny from 199.119.200.0/21

# AS5577 netdedicated.ru, root.lu
deny from 46.166.143.0/24
deny from 91.212.226.0/24
deny from 94.242.192.0/18
deny from 212.117.160.0/19

# AS197043 exetel.de
deny from 31.214.133.0/24
deny from 46.251.224.0/20
deny from 109.230.208.0/20
deny from 109.230.240.0/20

# solutionpro.com - Boise, Idaho
deny from 206.80.96.0/19
deny from 206.206.0.0/15
deny from 207.70.0.0/18
deny from 209.19.128.0/18

# AS41079 SuperHost.pl
deny from 178.250.40.0/21
deny from 193.218.152.0/22

# bluecoat.com
deny from 103.246.36.0/22
deny from 65.46.48.192/30
deny from 65.160.238.176/28
deny from 204.246.128.0/20
deny from 208.115.138.0/23
deny from 216.16.247.0/28
deny from 217.169.46.96/28

# peakwebhosting.com
deny from 204.11.216.0/21

# AS46475 limestonenetworks.com
deny from 64.31.0.0/18
deny from 69.162.64.0/18
deny from 74.63.192.0/18
deny from 208.115.192.0/18
deny from 216.245.192.0/19

# AS8737 planet.nl
deny from 77.166.0.0/16

# AS24961 fast IT Colocation fibre1.net
deny from 85.114.128.0/20

# AS11274 Adhost Internet Advertising, LLC adhost.com
deny from 67.212.128.0/20
deny from 96.31.160.0/20
deny from 173.240.48.0/20
deny from 199.21.64.0/21
deny from 199.193.176.0/22
deny from 216.182.80.0/20
deny from 216.211.128.0/20

# AS46844 sharktech.net
deny from 204.188.192.0/18

# AS13448 websense
deny from 66.194.6.0/24
deny from 67.117.201.128/28
deny from 91.194.158.0/23
deny from 192.132.210.0/24
deny from 204.15.64.0/21
deny from 208.80.192.0/21

# twtelecom.com, possibly used by websense
deny from 207.114.128.0/17

# savvis is the host for the attributor stealth bot
deny from 64.41.128.0/17

# cogentco.com - used by cyveillance stealth bot
deny from 38.0.0.0/8
deny from 65.213.208.128/27
deny from 65.222.176.96/27
deny from 65.222.185.72/29
deny from 151.173.0.0/16

# AS8551 bezeqint hosting
deny from 82.80.224.0/20
deny from 82.80.240.0/21
deny from 82.80.248.0/21

# AS3255 Ukrainian Academic and Research Network
deny from 192.162.19.0/24

# AS31148 FREENET dial-up service in New Zealand
deny from 89.252.58.0/24

# Silknet
deny from 149.3.87.0/24

# AS57232/AS15936 spammer in the Ukraine
deny from 82.193.96.0/19
deny from 91.231.40.0/22
deny from 109.106.0.0/19

# AS16276 OVH SAS ovh.com
deny from 5.135.0.0/16
deny from 37.59.0.0/16
deny from 46.105.0.0/16
deny from 87.98.128.0/17
deny from 91.121.0.0/16
deny from 94.23.0.0/16
deny from 192.95.0.0/18
deny from 192.99.0.0/16
deny from 176.31.0.0/16
deny from 178.32.0.0/15
deny from 188.165.0.0/16
deny from 213.186.32.0/19
deny from 213.251.128.0/18

# AS24940 Hetzner Online AG
deny from 176.9.0.0/16
deny from 46.4.0.0/16
deny from 5.9.0.0/16
deny from 78.46.0.0/15
deny from 88.198.0.0/16
deny from 144.76.0.0/16
deny from 148.251.0.0/16

# AS32875 Virpus Network Operations
deny from 199.180.128.0/21

# AS53264, AS15003 Continuum Data Centers
deny from 216.231.128.0/20

# compass communications: used by domaintools.com spider
deny from 216.145.0.0/19
deny from 64.246.160.0/19

# endurance international: this is an umbrella company for dozens of hosting companies
deny from 66.249.0.0/19

# random subnets that generate a lot of spam
deny from 176.119.0.0/24

# AS20473 choopa: network ahrefs bot runs under
deny from 64.237.32.0/19
deny from 66.55.128.0/19
deny from 68.232.160.0/19
deny from 108.61.0.0/16
deny from 173.199.64.0/18
deny from 208.167.224.0/19
deny from 209.222.0.0/19
deny from 216.155.128.0/19

# AS16124 utel datacenter networks
deny from 213.186.112.0/20

# AS58049 Telecom Tekhpodderzhka Ltd
deny from 91.237.249.0/24

# AS43391 Netdirekt: ISP in Turkey
deny from 77.223.128.0/19

# AS47143 todayhost: host of WBSearchBot
deny from 195.42.102.0/23

# AS44050 Russia IzydorSymanski-net
deny from 188.143.232.0/23

# Chinanet non-portable www.fjtelecom.com
deny from 220.160.0.0/15
deny from 220.162.0.0/16

# AS20860 IOmart Hosting
deny from 78.129.128.0/17
deny from 95.154.192.0/18

# AS16276 OVH
deny from 46.105.0.0/16

# AS10316 Codero codero.com
deny from 64.150.176.0/20
deny from 66.226.72.0/21
deny from 68.168.96.0/20
deny from 69.64.64.0/19
deny from 162.244.64.0/22
deny from 206.225.80.0/21
deny from 206.225.92.0/22
deny from 216.55.128.0/22
deny from 216.55.136.0/21
deny from 216.55.160.0/21
deny from 216.55.168.0/22
deny from 216.55.176.0/21
deny from 216.55.184.0/22

# AS29073 Ecatel Dedicated Hosting (Netherlands)
deny from 93.174.88.0/21

# rebelhosting.net
deny from 199.33.120.0/21

# AS12260 colostore.com
deny from 67.214.160.0/19

# AS30152 beyondhosting.net
deny from 8.29.136.0/21

# uk2group.com Hosting Services, Inc.
deny from 98.158.176.0/20

# AS13601 peer1.net
deny from 76.74.128.0/17
deny from 216.157.0.0/18
deny from 216.157.64.0/19
deny from 216.157.96.0/20

# AS51559 netinternet in Turkey
deny from 94.102.0.0/20
deny from 95.173.160.0/19

# sanitytechnology.com.au
deny from 103.1.216.0/22
deny from 122.99.112.0/21

# Japan tsukaeru.net
deny from 27.112.104.0/21
deny from 175.45.136.0/21

# VPNTUNNEL-INET vpntunnel.se
deny from 5.254.144.0/24

# AS8972 plusserver.de
deny from 188.138.0.0/17

# godaddy
deny from 50.62.0.0/15
deny from 216.69.128.0/18

# AS50673 serverius
deny from 46.249.32.0/19

# AS8708 RCS & RDS Business Romania
deny from 81.196.0.0/16

# scanner from evuln.com comes from this class C
deny from 78.158.11.0/24

# webair.com
deny from 174.137.128.0/18

# FiberMax USA
deny from 178.18.16.0/22

# AT&T enterprise ip group
deny from 192.20.240.0/20

# dacentec.com
deny from 199.241.184.0/21

# AS14061 digitalocean.com
deny from 208.68.36.0/22

# AS4436 nLayer nlayer.com
deny from 63.141.192.0/19
deny from 69.22.128.0/18
deny from 198.144.96.0/19
deny from 204.93.32.0/19

# AS19194 DME Hosting dmehosting.com
deny from 74.221.208.0/23

# netriplex.com aka microglobe
deny from 208.69.228.0/22
deny from 216.59.0.0/18

# AS39572 advancedhosters.com
deny from 46.229.160.0/20

# AS593-AP web24.com.au
deny from 111.67.0.0/19
deny from 125.214.64.0/20
deny from 203.16.60.0/23
deny from 223.27.0.0/19

# steadfast.net
deny from 50.31.0.0/17

# AS32475 singlehop.com
deny from 96.127.128.0/18
deny from 108.163.192.0/18
deny from 184.154.0.0/16
deny from 198.143.128.0/18

# continuumdatacenters.com
deny from 199.192.200.0/21

# wholesaleinternet.com
deny from 69.197.128.0/18
deny from 173.208.128.0/17
deny from 204.12.192.0/18
deny from 208.110.64.0/19

# iweb.com
deny from 209.172.32.0/19
deny from 64.15.128.0/19
deny from 72.55.128.0/18
deny from 67.205.64.0/18
deny from 70.38.0.0/17
deny from 174.142.0.0/16
deny from 184.107.0.0/16
deny from 198.72.96.0/19
deny from 198.50.96.0/19

# xlhost.com
deny from 64.79.64.0/19
deny from 173.45.64.0/18
deny from 173.244.160.0/19
deny from 207.182.128.0/19
deny from 209.190.0.0/17

# AS33139 canaca.com
deny from 173.248.192.0/18

# pegtechinc petaexpress.com
deny from 142.0.128.0/20
deny from 142.4.96.0/19
deny from 192.74.224.0/19
deny from 198.200.32.0/19
deny from 199.180.100.0/22

# crucialp.com, secureip.net.au
deny from 203.98.64.0/19

# areti internet, alentus.co.uk, alentus.com
deny from 85.237.192.0/19
deny from 204.14.72.0/21
deny from 216.185.32.0/19

# anchorfree.com web proxy
deny from 74.115.0.0/21
deny from 199.255.208.0/21

# fastq.com
deny from 65.39.64.0/19

# americanis.net
deny from 204.68.96.0/19

# AS44050 Russian data centre pinroute
deny from 31.184.238.0/24
deny from 46.161.41.0/24
deny from 188.143.232.0/23
deny from 188.143.234.0/23

# arrival.com
deny from 69.84.192.0/20

# colosseum.com
deny from 204.101.51.0/24

# fortinet.com
deny from 204.101.161.0/24

# virtbiz.com
deny from 208.77.216.0/21

# AS29017 gyron.net
deny from 89.145.64.0/18

# fdcservers.net
deny from 67.159.0.0/18
deny from 74.63.64.0/18

# tuliptel.com, tulip.net
deny from 110.234.0.0/15

# linode.com
deny from 69.164.192.0/19
deny from 162.216.16.0/22
deny from 192.81.128.0/21
deny from 192.155.80.0/20

# AS50245 serverel.com
deny from 173.214.240.0/20

# AS22439 perfectip.net
deny from 64.56.64.0/20

# AS40824 webazilla.com
deny from 199.80.52.0/22

# corenap.com
deny from 64.17.0.0/20
deny from 64.20.224.0/19
deny from 66.219.32.0/19
deny from 96.47.208.0/20
deny from 208.67.240.0/21
deny from 208.123.64.0/19

# AS24961 fastit.net, myloc.de
deny from 5.199.128.0/20

# AS11288 commercialmedia.com
deny from 198.74.228.0/22

# websitewelcome.com
deny from 192.254.128.0/17

# internap.com
deny from 212.118.224.0/19

# pacific.net.au
deny from 203.143.192.0/18

# custodiandc.com
deny from 5.102.168.0/21

# bahnhof.net
deny from 37.123.128.0/18

# singlehop.com
deny from 108.178.0.0/18

# source ranges of YisouSpider (ALISOFT)
deny from 42.156.128.0/17

# class Cs used by 360Spider, aka CollapsarTEXT, a content scraper from China
deny from 61.55.185.0/24
deny from 101.226.166.0/24
deny from 101.226.167.0/24
deny from 101.226.168.0/24
deny from 101.226.169.0/24
deny from 180.153.236.0/24
deny from 182.118.20.0/24
deny from 182.118.21.0/24
deny from 182.118.22.0/24
deny from 182.118.25.0/24
deny from 182.118.26.0/24
deny from 182.118.35.0/24
deny from 182.118.55.0/24
deny from 182.118.56.0/24

# alestra.net.mx
deny from 201.151.0.0/16

# inlink communications: primary.net
deny from 206.196.96.0/19

# AS33926 kaiaglobal.com
deny from 79.141.160.0/20
deny from 95.141.16.0/20
deny from 185.18.104.0/22
deny from 193.34.48.0/22
deny from 193.142.97.0/24
deny from 194.124.229.0/24
deny from 195.13.60.0/22
deny from 195.78.240.0/22

# resilans.se
deny from 192.121.0.0/16
deny from 193.180.0.0/16

# edis.at
deny from 151.236.19.0/24

# slaskdatacenter.pl
deny from 178.19.104.0/21

# colocrossing.com
deny from 23.94.0.0/15
deny from 75.127.0.0/20
deny from 96.8.112.0/20
deny from 107.172.0.0/14
deny from 108.174.48.0/20
deny from 172.245.0.0/16
deny from 192.3.0.0/16
deny from 192.210.128.0/17
deny from 192.227.128.0/17
deny from 198.12.64.0/18
deny from 198.23.128.0/17
deny from 198.46.128.0/17
deny from 198.144.176.0/21
deny from 199.21.112.0/22
deny from 199.188.100.0/22
deny from 206.217.128.0/20

# gnax.net
deny from 75.127.64.0/18

# he.net
deny from 65.49.0.0/17

# amanah.com
deny from 184.75.208.0/20

# centarra.com / avante
deny from 198.52.128.0/17

# dacentec.com
deny from 23.92.208.0/20
deny from 162.248.240.0/21
deny from 192.111.144.0/20
deny from 192.198.80.0/20
deny from 192.254.64.0/20
deny from 199.101.184.0/22
deny from 199.191.56.0/22
deny from 199.241.184.0/21
deny from 199.255.156.0/22

# quadranet.com
deny from 66.63.160.0/19
deny from 69.12.64.0/19
deny from 72.11.128.0/19
deny from 96.44.128.0/18
deny from 98.143.144.0/20
deny from 107.161.80.0/20
deny from 173.254.192.0/18
deny from 192.161.48.0/20
deny from 192.161.160.0/19
deny from 198.55.96.0/19
deny from 198.96.88.0/21
deny from 204.44.64.0/18
deny from 204.152.192.0/19
deny from 216.45.48.0/20

# krypt.com, vpls.net, kryptcolo.net
deny from 66.186.32.0/19
deny from 67.198.128.0/17
deny from 67.229.0.0/16
deny from 74.222.128.0/18
deny from 96.62.0.0/16
deny from 98.126.0.0/16
deny from 100.43.128.0/18
deny from 107.6.192.0/18
deny from 173.214.0.0/17
deny from 174.139.0.0/16
deny from 184.75.176.0/20
deny from 184.83.0.0/16
deny from 184.164.192.0/19
deny from 192.174.96.0/19
deny from 209.11.240.0/20

# gorillaservers.com
deny from 23.29.64.0/20
deny from 23.239.96.0/19
deny from 107.181.224.0/19
deny from 173.231.60.0/23
deny from 192.154.96.0/20
deny from 192.200.96.0/19
deny from 198.100.96.0/19
deny from 198.136.24.0/21
deny from 199.73.120.0/22

# google hosting (GAE)
deny from 8.35.200.0/21
deny from 107.178.192.0/18

# class C blocks that contain ip anonymizers
deny from 41.190.88.0/24
deny from 41.215.160.0/24

# Various problematic class Cs from China
deny from 1.202.219.0/24
deny from 14.17.18.0/24
deny from 14.17.29.0/24
deny from 14.17.34.0/24
deny from 42.62.37.0/24
deny from 101.226.33.0/24
deny from 101.226.51.0/24
deny from 101.226.65.0/24
deny from 101.226.66.0/24
deny from 101.226.89.0/24
deny from 101.226.102.0/24
deny from 110.75.171.0/24
deny from 110.75.172.0/24
deny from 110.75.173.0/24
deny from 110.75.174.0/24
deny from 110.75.175.0/24
deny from 111.161.56.0/24
deny from 111.161.62.0/24
deny from 112.64.235.0/24
deny from 112.65.193.0/24
deny from 113.108.81.0/24
deny from 125.39.240.0/24
deny from 180.153.114.0/24
deny from 180.153.160.0/24
deny from 180.153.161.0/24
deny from 180.153.163.0/24
deny from 180.153.201.0/24
deny from 180.153.205.0/24
deny from 180.153.206.0/24
deny from 180.153.211.0/24
deny from 180.153.212.0/24
deny from 180.153.213.0/24
deny from 180.153.214.0/24
deny from 180.153.240.0/24
deny from 220.181.158.0/24
deny from 222.73.76.0/24
deny from 222.73.77.0/24

# visafone ISP in Nigeria - source of scammers likely using vpnsecure.me
deny from 41.138.160.0/19
deny from 41.71.171.0/17

# spectranet ISP in Nigeria - source of scammers hiding behind anonymizers
deny from 197.242.0.0/16
deny from 197.255.0.0/16

# multilinks.com - Multilinks Telecommunications in Nigeria - source of scammers hiding behind anonymizers
deny from 82.128.0.0/20

# airtel.com - Nigerian scammers
deny from 196.46.240.0/21

# Etisalat Nigeria - more scammers
deny from 41.190.0.0/19

# MY ADDITION
deny from 180.76.5.0/24
deny from 180.76.6.0/24
deny from 123.125.71.0/24
deny from 220.181.108.0/24
deny from 119.63.192.0/21

# deny user-agents
deny from env=bad_bot

# allow everyone else in
allow from all

</Location>

[Mark_Young]

user-agent_blacklist.conf

SetEnvIfNoCase User-Agent "rogerbot" bad_bot
SetEnvIfNoCase User-Agent "exabot" bad_bot
SetEnvIfNoCase User-Agent "MJ12bot" bad_bot
SetEnvIfNoCase User-Agent "dotbot" bad_bot
SetEnvIfNoCase User-Agent "gigabot" bad_bot
SetEnvIfNoCase User-Agent "AhrefsBot" bad_bot
SetEnvIfNoCase User-Agent "nerdybot" bad_bot
SetEnvIfNoCase User-Agent "baidu" bad_bot
SetEnvIfNoCase Referer "semalt" bad_bot
SetEnvIfNoCase Referer "yandex" bad_bot
SetEnvIfNoCase Referer "pricingprotection" bad_bot

---

and the top of my httpd.conf file showing how I have included them:

Include "/usr/local/apache/conf/includes/pre_main_global.conf"
Include "/usr/local/apache/conf/includes/pre_main_2.conf"



LoadModule bwlimited_module modules/mod_bwlimited.so
LoadModule cloudflare_module modules/mod_cloudflare.so




Include "/usr/local/apache/conf/includes/user-agent_blacklist.conf"
Include "/usr/local/apache/conf/php.conf"
Include "/usr/local/apache/conf/includes/ip_blacklist.conf"
Include "/usr/local/apache/conf/includes/errordocument.conf"
Include "/usr/local/apache/conf/modsec2.conf"


ErrorLog "logs/error_log"
ScriptAliasMatch ^/?controlpanel/?$ /usr/local/cpanel/cgi-sys/redirect.cgi
ScriptAliasMatch ^/?cpanel/?$ /usr/local/cpanel/cgi-sys/redirect.cgi
ScriptAliasMatch ^/?kpanel/?$ /usr/local/cpanel/cgi-sys/redirect.cgi
ScriptAliasMatch ^/?securecontrolpanel/?$ /usr/local/cpanel/cgi-sys/sredirect.cgi
ScriptAliasMatch ^/?securecpanel/?$ /usr/local/cpanel/cgi-sys/sredirect.cgi
ScriptAliasMatch ^/?securewhm/?$ /usr/local/cpanel/cgi-sys/swhmredirect.cgi
ScriptAliasMatch ^/?webmail/?$ /usr/local/cpanel/cgi-sys/wredirect.cgi
ScriptAliasMatch ^/?whm/?$ /usr/local/cpanel/cgi-sys/whmredirect.cgi

RewriteEngine on
AddType text/html .shtml

Alias /bandwidth /usr/local/bandmin/htdocs/
Alias /img-sys /usr/local/cpanel/img-sys/
Alias /java-sys /usr/local/cpanel/java-sys/
Alias /mailman/archives /usr/local/cpanel/3rdparty/mailman/archives/public/
Alias /pipermail /usr/local/cpanel/3rdparty/mailman/archives/public/
Alias /sys_cpanel /usr/local/cpanel/sys_cpanel/


ScriptAlias /cgi-sys /usr/local/cpanel/cgi-sys/
ScriptAlias /mailman /usr/local/cpanel/3rdparty/mailman/cgi-bin/


<Directory "/">
 AllowOverride All
 Options ExecCGI FollowSymLinks IncludesNOEXEC Indexes SymLinksIfOwnerMatch
</Directory>

<Directory "/usr/local/apache/htdocs">
 Options All
 AllowOverride None
 Require all granted
</Directory>

<Files ~ "^error_log$">
 Order allow,deny
 Deny from all

 Satisfy All
</Files>

<Files ".ht*">
 Require all denied
</Files>

<IfModule log_config_module>
 LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
 LogFormat "%h %l %u %t \"%r\" %>s %b" common

 CustomLog "logs/access_log" common

 <IfModule logio_module>
 LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

 </IfModule>

</IfModule>

<IfModule alias_module>
 ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin/"

</IfModule>

<Directory "/usr/local/apache/cgi-bin">
 AllowOverride None
 Options All
 Require all granted
</Directory>

<IfModule mime_module>
 TypesConfig conf/mime.types
 AddType application/x-compress .Z
 AddType application/x-gzip .gz .tgz

</IfModule>

<IfModule prefork.c>
 Mutex default mpm-accept

</IfModule>

<IfModule mod_log_config.c>
 LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvhost
 LogFormat "%v %{%s}t %I .\n%v %{%s}t %O ." bytesvhost
 LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
 LogFormat "%h %l %u %t \"%r\" %>s %b" common
 LogFormat "%{Referer}i -> %U" referer
 LogFormat "%{User-agent}i" agent

 CustomLog "|/usr/local/cpanel/bin/splitlogs --main=cpanel.server.zz --suffix=-bytes_log" bytesvhost
 CustomLog "|/usr/local/cpanel/bin/splitlogs --main=cpanel.server.zz --mainout=/usr/local/apache/logs/access_log" combinedvhost

</IfModule>

<IfModule itk.c>
 Mutex default mpm-accept

</IfModule>

<IfModule worker.c>
 Mutex default mpm-accept

</IfModule>

[wilderness]

FYI

Forum Charter [webmasterworld.com]

Forum Etiquette:
It is not appropriate to expect other members to write your code for you or to debug your entire project; Please don't expect other members to solve a problem you don't want to begin solving yourself.

We understand that it is often necessary to provide code samples. Just keep them short and focused please; Excessive code dumps will be edited or deleted.

[Mark_Young]

Hi wilderness,

I appreciate your post and for pointing out the Forum Charter.

My intentions were to give as much information as possible. I thought this would aide anybody going out of their way to try to help me or point me in the right direction. I wrongly presumed too much information was better than not enough.

Maybe I could break the post down to:

I have a large list of IP's and user-agents I want to block from my server. What is the best way to do this server wide so that the blocks apply to any websites hosted on the server - maybe there is a third party script that aleady achieves this that includes a easy way to manage/update the block lists?

Thank you for any help and please let me know if you require any further information to point me in the right direction.

[wilderness]

I'm not an Apache person, rather I've been dealing with primarily htaccess on shared hosts for fourteen years.

There are some Apache folks here whom will chirp in eventually.

I have a large list of IP's and user-agents I want to block from my server. What is the best way to do this server wide so that the blocks apply to any websites hosted on the server

A config file for the entire server should serve you purpose.

maybe there is a third party script that aleady achieves this that includes a easy way to manage/update the block lists?

You will have to adapt these to your config file.

This method by keymaster [webmasterworld.com] is still effective.

A bit newer version by keymaster [webmasterworld.com]


There are two things (organizational) that you could do to make keeping track of your htaccess/config easier and more efficient.

1) you have your IP's grouped and commented by name. It's easier to maintain when they are organized sequentially.

1a) You may insert multiple CIDR's on lines.
EX: (I've lines much longer than this, however would suggest some uniform/consistent limit on line-length):

Deny from 67.80.24. 67.82.208. 67.83.157. 67.86.177. 67.93.234.

2) UA's may also be listed sequentially and with many on a single line in mod-rewrite.
EX (I make every attempt to limit these to a 6-8 line limit):

RewriteCond %{HTTP_USER_AGENT} ^(deepak|devSoft|Diamond|Digger|dloader) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} c(entre|erberian|lient|rawl|sci|url) [NC,OR]

These are just examples to assist you.

[not2easy]

I think if you are using the block list as is, you may just have the server dropping the ball before being able to completely check every visit against that list. It is very inefficient for the server to check against 3 or 4 CIDRs and then start over. As wilderness said, one long list in sequence is far more useful that that broken up list.

I also only work with .htaccess files and block a lot of server scum so I can't really help about the proper format for httpd.conf but if blocked entities are getting through I would guess that your process is just timing out. That is a very difficult list to use in that format. It essentially forces the server to process each visitor several dozen times, starting at .1 each time.

[lucy24]

Are you asking about the exact content of your block list, or about the "Include" mechanism? Your initial post made it sound as if the question was about the mechanics of the "Include" locution in a config file-- a detail that should be easy to test by simply including a file that blocks yourself and verifying that it works. But if so, the question is all but buried under line after line of Deny from... directives. People are liable to get exhausted before they even figure out the question :(

Personally I think it makes no sense to classify the IP list so precisely. The server doesn't care, but you'd spend half your time just checking whether some value is on the list already. Just put them all in numerical order and be done with it. At most, sort them into RIPE and ARIN if that turns out to be more convenient.

[Mark_Young]

Thnak you for all replies, it's appreciated.

It seems the method I am using for blocking is correct, it's just the layout that's the problem (and the poor layout of the list may be the reason some IP's in the blocklist are missed)?

As an example, are you saying my IP blocklist file should look something like this:

<Location />

order allow,deny

deny from 1.202.219.0/24 14.17.18.0/24 14.17.29.0/24 14.17.34.0/24 42.62.37.0/24 101.226.33.0/24 101.226.51.0/24 101.226.65.0/24 101.226.66.0/24 101.226.89.0/24 101.226.102.0/24 110.75.171.0/24 110.75.172.0/24 110.75.173.0/24 110.75.174.0/24 110.75.175.0/24 111.161.56.0/24 111.161.62.0/24 112.64.235.0/24 112.65.193.0/24 113.108.81.0/24 125.39.240.0/24 180.153.114.0/24 180.153.160.0/24 180.153.161.0/24 180.153.163.0/24 180.153.201.0/24 180.153.205.0/24 180.153.206.0/24 180.153.211.0/24 180.153.212.0/24 180.153.213.0/24 180.153.214.0/24 180.153.240.0/24 220.181.158.0/24 222.73.76.0/24 222.73.77.0/24

RewriteCond %{HTTP_USER_AGENT} ^(deepak|devSoft|Diamond|Digger|dloader) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} c(entre|erberian|lient|rawl|sci|url) [NC,OR] 

allow from all

</Location> 

Does that look about right?

Thank you!

[wilderness]

these lines are going to expand and you need to break them down smaller into a consistent-LIMIT-length defined by yourself.

EX:
deny from 1.202.219.0/24 14.17.18.0/24 14.17.29.0/24 14.17.34.0/24 42.62.37.0/24
deny from 101.226.33.0/24 101.226.51.0/24 101.226.65.0/24 101.226.66.0/24 101.226.89.0/24 101.226.102.0/24
deny from 110.75.171.0/24 110.75.172.0/24 110.75.173.0/24 110.75.174.0/24 110.75.175.0/24 111.161.56.0/24 111.161.62.0/24 112.64.235.0/24 112.65.193.0/24 113.108.81.0/24
deny from 125.39.240.0/24
deny from 180.153.114.0/24 180.153.160.0/24 180.153.161.0/24 180.153.163.0/24 180.153.201.0/24 180.153.205.0/24 180.153.206.0/24 180.153.211.0/24 180.153.212.0/24 180.153.213.0/24 180.153.214.0/24 180.153.240.0/24
deny from 220.181.158.0/24 222.73.76.0/24 222.73.77.0/24

this makes the entire syntax more manageable and easier to navigate.

[wilderness]

The following lines do not go into the same section and require additional syntax ( I merely provided them as an example of how to organize UA's):

RewriteCond %{HTTP_USER_AGENT} ^(deepak|devSoft|Diamond|Digger|dloader) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} c(entre|erberian|lient|rawl|sci|url) [NC,OR]

the [OR] is applicable to lines in succesion, however the last line of each such succession grouo is void the [OR] else you'll get a 500 error taking the entire server/site down.

IF these were the only two lines?

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^(deepak|devSoft|Diamond|Digger|dloader) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} c(entre|erberian|lient|rawl|sci|url) [NC]
RewriteRule .* - [F]

I'e two particular sections in mod-rewrite dealing with the UA's (as per the examples provided).

One group if for ^"begins with".
The other group for (note; absent any anchor)"contains"


The understanding of anchors and their use is crucial and basic fundamentals.

Begins with
Ends with
Begins and ends with
Exactly as
Contains

[lucy24]

RewriteCond %{HTTP_USER_AGENT} ^(deepak|devSoft|Diamond|Digger|dloader) [NC,OR]

RewriteCond %{HTTP_USER_AGENT} c(entre|erberian|lient|rawl|sci|url) [NC,OR]

This kind of thing doesn't need to be a RewriteRule. Use BrowserMatch (or, if absolutely necessary, BrowserMatchNoCase) in mod_setenvif to define something like "keep_out" or "bad_bot". Then add the single line

Deny from env=keep_out

and you don't have to worry about inheritance.

I don't think lockouts need to be in any kind of envelope, and certainly not <Location>. You want to bar them from the whole server, including any and all sites, don't you?

[wilderness]

Use BrowserMatch

Once again, I'll never understand this fascination, with something that has such negative effect.

The following single line and some subsequent similar lines, overrides the entire rest of the htaccess file.

BrowserMatchNoCase ^Mozilla good_pass

[lucy24]

The following single line and some subsequent similar lines, overrides the entire rest of the htaccess file.

Huh? Each module issues its own 403s; no mod can cancel a 403 issued by any other mod. Besides, mod_setenvif in and of itself doesn't issue 403s. You need some other mod to read the environmental variable. mod_authz-thingy is the easiest (i.e. least work on the server).

A form like like "good_pass" only has meaning if you whitelist by UA. You could do the same in blacklisting mode by un-setting a negative environmental variable-- but the situations where you'd do so are rare. (Currently the only place I do it is to keep from having to manually unlock the door for the link checker every time I use it.)

Besides, nobody in their right mind would say NoCase when matching against the case-specific form "Mozilla" ;) If it says "mozilla" or "MOZILLA" it's fake, just as "GoogleBot" is fake by definition.