Forum Moderators: phranque
To avoid this in the future I plan on building some sort of system that will count the number of times an IP hits the site & after so many hits in a specified period of time it'll redirect to a static error page (which most of my pages are perl/mysql & that's why the server went down).
A couple questions on this:
Is there something already out there (that is free) that I could use to stop this type of attack (that I can use on a shared hosting account)?
If not, what would be the best setting like x hits every x seconds so I don't deflect search engines for my sites. For the most part I would want to allow all search engines unlimited access (except maybe gigabot: ate up 2GB bandwidth in one afternoon & no benefits that I've seen).
I would just like some input from some people who have dealt with this. I don't even know if my idea is practical: logging every hit to every page of my websites & checking data against the timestamps of identical IP addresses & calculating the amount of hits within a speified period of time for that IP address - Maybe that'll cause too much stress on the server in itself? At the moment my busiest site gets about 3,000 unique visits a month - the rest of the sites are less than that.
Any input on this would be appreciated.
Your options depend on your server platform, ditto your webserver -- and how much your SysAdmins want to mess with things -- and whether what they may install server-wide can be tailored to individual accounts on the server -- and --
And that's a lot of significant ifs, ands, and whats.
However, if you're running Apache, check out "mod_bwshare" as well as the various modules I mentioned here [webmasterworld.com] (or forward that post to your host). I spent ages and ages searching for ways to achieve similar goals and that list, and that mod, were most appropriate for my Wish List and my sites (all of which sound similar to yours, except that I own the Linux box and the SysAdmin is my husband:)
I only wish I could tell you that "mod_bwshare" or any throttling module worked like a charm! Alas, we've yet to be able to compile that one -- and neither, apparently, has anyone else here (no replies to my post) -- and the others just don't fit in well with what I really need. Good luck!
The most recent of a 3-part thread, with lots of discussion. Though aimed at robots, it would work for DOS attacks as well.
Jim
Like serveral other WMW members, I coded my own solution, though there are now standard programs you can download.
You don't need to keep a large amount of data to do this. The way I do it is to look in a 3 minute window.
Any IP address that occurs more than 20 times in one minute or 30 times in 2 minutes or 35 times in 3 minutes is assumed to be anti-social bot.
An anti-social bot has its IP address banned for 1 hour.
I check the ban log every few days and if there is a consistent offender, I morph their IP address into a permanent ban via an entry a .htaccess entry.
The worst a bot can do is hit me 20 times in a minute, once an hour for a couple of days.
(The actual times I use are different, but you get the idea).
Pfui, I checked out your post & I did like the mod_bwshare mod (yes, I'm on apache). If I were on a dedicated server I'd probably implement that, but I'd rather not have my host have to mess with anything on my account right now especially when I had to almost beg them to keep me hosted there. (the attacked domain was removed, the others stayed) Though I think they can make changes for each account that would not be server wide.
Jim, That contains so much info. I read through most of part 3 and will look through the other parts as well as I find time here shortly. Thanks for that link, I pulled quite a few ideas off just the parts I read.
Victor, I think that I, too, will create my own program. I knew that I didn't need to store too much info as I would be banning people/bots for so many hits in so many minutes (not hours) but was concerned about the repeated checking of the ip address mixed in with the other dynamic content on my sites. Also - thanks for the rough ideas of timeframes for blocking.
I'll still search on some server flood control ideas that I can implement without root access. I know that multiple hits from the same ip address is not the only way that someone can cause issues. For instance, if site A is a very high traffic site & they didn't like site B for some reason site A could just use an iframe to load a resource intensive page on site B every time they get a visitor. . . and ip banning wouldn't help out in that situation.
But we'll just take it step by step & deal with this part for now & deal with other attack prevention methods as time allows (or necessity dictates).
Thanks again for all the input, it has been very helpful.
