I've recently had a huge crawl by a number of spambots on my sites. I need to start blocking them.. Wanted to consult with you about it.
A few years ago, on a java based project I was involved at, we solved this by monitoring number of requests per minutes of certain pages, and for every unfriendly useragent / ip, if the number of requests per min met a certain threshold, it was assumed that this host is a hostile bot- and a the host was blocked with a captcha page- the host was unblocked only when it passed the captcha test.
Now, I'm working oh php platform, and I wouldnt want to go through the hassle of re-developing the entire mechanism in php- plus, since it's been a few years, i thought that something like this must exist :-)
I wanted to ask if anyone knows on an apache module / script that does something similar to a site-
ie- identify hostile bots, and preset them with captcha tests or otherwise block them.