Monitoring bot activity with a script?

[Bluestreak]

Hi all,

I maintain a ban list using mod_rewrite to prevent malicious and poorly designed bots from entering the site. However, Im looking for a proactive solution as well that would keep me apprised of bot activities as they happen.

Is it possible to have a script that monitors bots that come to the site, and initiates an automatic IP ban when the bot ignores robots.txt or goes whacky? It would be a nice solution that wouldnt require you to check your logs every five seconds for any malicious bot activities, and manually place the offenders on a ban list. Instead, a script could automate the process, creating a dynamic ban list for misbehaving bots.

Are there any scripts out there that can accomplish something similar to this, or am I thinking slightly ahead of the times?

Thanks for any advice or help!

[wilderness]

This thread will help some
[webmasterworld.com...]

[Bluestreak]

Thanks for the thread link. It has the right idea, but the only thing I saw was mention of a perl module that could be used. I dont have direct access to the server, so I cant install modules at whim. Seems like a lot of users talk about how they set up bottraps utilizing scripts but I can never seem to find them. Are there any scripts available either commercially or in the public domain that would help me in setting up a spidertrap?

[wilderness]

[google.netscape.com...]

[Bluestreak]

Thanks for the link. Ive already searched through Google using other keywords, but there seems to be a real lack of pre made scripts for those less technically inclined like me. Most of what I found were tutorials or people making vague references to how they coded a script to trap a bot using a honeypot directory, but evidently nothing is available for public use, commercial or otherwise.

Since this is an important issue regarding web security and protection, why are there so few scripts available?

[Key_Master]

Maybe this [neilgunton.com] can be of some help. Sorry, most anything to do with a scripting language is going to sound technically challenging at first. It's intimidating at first but with a little experience, you'll find it hard to imagine not using it.

[wilderness]

KeyMasters link excellent.

The search I pviously suuplied offers the first hit. Which contains a link to a non active page which you might consider looking at.
Whether the links are still active or not?http://web.archive.org/web/20011217181645/http://www.spiderhunter.com/

[Bluestreak]

Thanks for the help. I appreciate the responses! Ill check out the links and see what I can find.

I'm not TOO intimidated at looking at the scripts. After dealing with PHP based scripts, I've learned a lot and have actually on occasion been able to debug certain aspects of PHP code on my site that werent functioning properly. Creating an entirely NEW script from scratch is a totally different story though :-) Im hoping to avoid that if I can.

With my current setup, I have a honeypot directory that is expressly forbidden in my robots.txt file. My mission is to find or create a script that will monitor if that directory had been accessed by a bot that completely ignored robots.txt and to initiate a ban of that bot's user agent or IP address. Once I get that, I'm good to go :D