homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
Forum Library, Charter, Moderators: coopster & jatar k

PHP Server Side Scripting Forum

blocking robots using apache / php

 9:26 am on Jan 22, 2007 (gmt 0)

Hi All,

I've recently had a huge crawl by a number of spambots on my sites. I need to start blocking them.. Wanted to consult with you about it.
A few years ago, on a java based project I was involved at, we solved this by monitoring number of requests per minutes of certain pages, and for every unfriendly useragent / ip, if the number of requests per min met a certain threshold, it was assumed that this host is a hostile bot- and a the host was blocked with a captcha page- the host was unblocked only when it passed the captcha test.

Now, I'm working oh php platform, and I wouldnt want to go through the hassle of re-developing the entire mechanism in php- plus, since it's been a few years, i thought that something like this must exist :-)

I wanted to ask if anyone knows on an apache module / script that does something similar to a site-
ie- identify hostile bots, and preset them with captcha tests or otherwise block them.

many thanks!



 11:02 am on Jan 22, 2007 (gmt 0)

Well you could try to add something like the following to your htacces page

RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [NC]
RewriteRule ^.*$ - [F]

Take notice of the OR in the first line.
All bots you block, must have the [NC, OR] at the end, except the last one.

Now all you must do is to find a list of bots to block.. just g for this and you will get many info.

Best of luck.


 11:14 am on Jan 22, 2007 (gmt 0)

a mechanism based on the number of requests per timeframe and implemented in PHP for apache/unix is described here:


Kind regards,

Global Options:
 top home search open messages active posts  

Home / Forums Index / Code, Content, and Presentation / PHP Server Side Scripting
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved