homepage Welcome to WebmasterWorld Guest from 54.226.173.169
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Visit PubCon.com
Home / Forums Index / Code, Content, and Presentation / Apache Web Server
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL & phranque

Apache Web Server Forum

    
Dynamic Blocking of IPs
Having issues with DOS that I can't combat with mod_rewrite
Canton




msg:4398214
 2:06 pm on Dec 15, 2011 (gmt 0)

Hello all - I've been reading this forum for the past few days (as well as about a year ago) to learn more about blocking user agents and IPs using mod_rewrite. I've been largely successful for the past year, but now I'm facing an issue that is outside my realm of understanding.

Without going into too much detail - we have many sites, each with many pages, with each of those pages interacting with a database (reads/writes) for each request.

Typically this all works very well, but rogue user agents will occasionally come in and make outrageous numbers of requests. Not too big of a deal, as I can block user agents pretty easily. This does result in what I refer to as a "de facto DOS", i.e. - no one is trying to take down our server, they're just trying to scrape content or learn more about our sites.

Now, however, I'm having issues with requests that look "normal" in the sense that they are set up to look like it's coming from a normal user/browser (no referrer, but user agent is Mozilla, in this case). Can't block this because it's identical - as far as I can tell - to a request for a bookmarked page or type-in traffic.

For the moment, I'm manually adding IPs to my mod_rewrite rules to block these as well, but I don't want my job to be playing with user agents and IP lists every day.

So, to come to the point - I would like to use script of some sort that will block requests according to a set of rules. Basically a simple if/then statement. For example, last night the IP in question resulted in 17,000+ requests in under 3 minutes. Simple scenario would be - [if "X-number" of requests from IP (to "C" block) within "X-seconds" time period, do 403].

My question - I am I thinking in the right direction here? This is outside my standard realm of experience/knowledge, and I may be reinventing the wheel here. Or, I could have devised the proper solution...thoughts/recommendations?

Quick note add: just came across iptables for Linux. We're running Apache on FreeBSD 7.3.

Thanks!
~Canton

 

DeeCee




msg:4398226
 2:45 pm on Dec 15, 2011 (gmt 0)

See for example

[webmasterworld.com...]

Canton




msg:4398236
 3:08 pm on Dec 15, 2011 (gmt 0)

Thanks DeeCee - downloaded the code for that and reading the threads now. Also, it appears for BSD that IPFW/IPF/PF are options, though I have A LOT more reading to do on those before I can feel comfortable!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Code, Content, and Presentation / Apache Web Server
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved