blocking robots using apache / php

[flex55]

Hi All,

I've recently had a huge crawl by a number of spambots on my sites. I need to start blocking them.. Wanted to consult with you about it.
A few years ago, on a java based project I was involved at, we solved this by monitoring number of requests per minutes of certain pages, and for every unfriendly useragent / ip, if the number of requests per min met a certain threshold, it was assumed that this host is a hostile bot- and a the host was blocked with a captcha page- the host was unblocked only when it passed the captcha test.

Now, I'm working oh php platform, and I wouldnt want to go through the hassle of re-developing the entire mechanism in php- plus, since it's been a few years, i thought that something like this must exist :-)

I wanted to ask if anyone knows on an apache module / script that does something similar to a site-
ie- identify hostile bots, and preset them with captcha tests or otherwise block them.

many thanks!

[omoutop]

Well you could try to add something like the following to your htacces page

RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [NC]
RewriteRule ^.*$ - [F]

Take notice of the OR in the first line.
All bots you block, must have the [NC, OR] at the end, except the last one.

Now all you must do is to find a list of bots to block.. just g for this and you will get many info.

Best of luck.

[Romeo]

a mechanism based on the number of requests per timeframe and implemented in PHP for apache/unix is described here:

[webmasterworld.com...]
[webmasterworld.com...]

Kind regards,
R.