maximillianos - 4:31 am on Feb 26, 2011 (gmt 0)
I run a cron job every 5 min to check the latest log file entries. I export the results to a perl script that then does a lookup on the ip of any infringers. At that point if the infringer is not in my allowed list of bots, they get banned through APF for 4 hours.
This seems to keep the aggressive ones at bay. They get a handful of pages before my script snags them. if they come back 4 hours later they just get a handful more.
Every 4 hours I clear the ip blocks. Less risk of blocking good traffic down the road when an ip gets reassigned. It happens. I've seen it.