.htaccess Size and Server Performance

[BohrMe]

I'm developing a perl script that downloads a file of banned IPs and modifies my .htaccess file to deny access to those IPs. It's really pretty quick when you consider that the file contains over 27000 banned IPs and grows by a couple of hundred IPs everyday. But, I don't know what kind of performance hit I may encounter as a result.

The person who generates this file uses the information in his /etc/hosts.deny file and doesn't notice a slow down and he gets ~3 million page hits a month. My site, on the other hand, is on a remote server where I can only manipulate the .htaccess.

Does anyone have an idea?

Thanks.

p.s., if you want the URL for the ban list, private message me or if the moderator permits, I could post it here. It is publically available.

[jdMorgan]

BohrMe,

Welcome to WebmasterWorld [webmasterworld.com]!

I'll offer an opinion that 27,000 IP addresses is way too many to put into .htaccess, unless you can sort them, de-duplicate them, and then test for address ranges to block instead of individual addresses.

Huge lists like this are much better handled by hardware using a content-addressable memory (CAM), where all blocked IPs can be checked in parallel, such as in higher-end firewalls.

From a practical point of view, you'd be better off to get a 'graded list' of bad IP addresses, and block the worst 100 maximum. Another alternative is to install a bad-bot trap script [webmasterworld.com] on your site and let it catch intruders and add their IP addresses to your .htaccess file. Then go through the list in .htaccess and your log files once week and trim the list back based on the severity of the attack that was blocked.

In other words, stop the actual attacks on *your* site without having to block every IP address that has ever been used for malicious purposes. When IPV4 rolls out, there will be 2^64 IP addresses, or about 4 billion times more IP addresses than we have today. Clearly, blocking *all* "bad" IP addressses will become impossible using serial comparison methods -- and quite expensive using content-addressable memory -- at that point.

The above is just my opinion, your mileage may vary. :)

Jim

[BohrMe]

The sort and de-duplication part is a no brainer but testing address ranges might prove a bit more difficult. Right now, the entire download, parse, sort, de-dupe and .htaccess rewrite takes a few seconds. On my site (a PHP-Nuke site), I have 24000+ IPs in a MySQL database that Protector System uses to filter users. It even has auto-banning, which has already paid-off by catching two different SQL injection exploits. I saw abolutely no decrease in performance when this was implemented. It was tested using different incoming IPs.

CAMs are great but expensive. I would love to have control over the hardware setup but off-site web hosting is much cheaper and my site generates no money.

I could trim it based on a two or three month history since most IPs out there are DHCP assinged addresses and will have changed since then. The current list goes back to 6-11-03. I have already caught some of those IPs attacking my site.

Most of the IPs listed in that ban list are people who have tried to sneak in port 445, 135, 57, 1080, 121, 500, 901, etc. Basically people who have port scanned his server.

I have already implemented the bad-bot trap posted here. That's very clever!