Safely banning bad bots with .htaccess

A few questions I can't find the answers to

[MatthewHSE]

Okay, so I've read through several of the .htaccess ban list [alltheweb.com] threads and have a pretty good idea of what's going on. But I do have a few questions that I can't find specifically addressed anywhere, and since I don't want to create a mess that I'll be sorry for later, I wanted some clarification before I begin.

• How often should the .htaccess ban list be updated?
  
• What are the consequences of not blocking bad bots?
  
• I don't understand just how I should handle the redirects. Some say to give a 403, others seem to prefer a script that e-mails them when the bot has been blocked. Some even mentioned ways of letting the bad bots reveal themselves and automatically add them to the list to block. What are the pros and cons to each method, and is there an all-on-one-page tutorial anywhere on the subject?

Any insights for a total beginner will be very much appreciated!

Thanks,

Matthew

[jdMorgan]

How often should the .htaccess ban list be updated?

As often as you need to -and can- update it.

What are the consequences of not blocking bad bots?

E-mail spam, copyright-legal hassles (founded and unfounded), site scrpers [webmasterworld.com], hotlinking, hijacking, anything bad, really.

I don't understand just how I should handle the redirects. Some say to give a 403, others seem to prefer a script that e-mails them when the bot has been blocked. Some even mentioned ways of letting the bad bots reveal themselves and automatically add them to the list to block. What are the pros and cons to each method, and is there an all-on-one-page tutorial anywhere on the subject?

No all-in-one that I know of. Handle the redirects in whatever way you want; Obviously, if you get 100,000 uniques a day, you sure don't want your script e-mailing you every time a new IP is blocked. I'd suggest you start with a short block list using a simple 403 at first, and then you can decide on further steps. I use all of the methods described in the thread linked above. Interestingly, having blocked most malicious activity, the number of attempts has decreased dramatically.

Jim

[moltar]

How often should the .htaccess ban list be updated?

My script does it automatically as soon as a bot steps in a trap.

I think a more important question is when do you remove the offending bot.

If you have a website with many visitors you might catch 10s if not 100s of bots each day. Your .htaccess will get out of control pretty fast. It will slow down the server, because the server will have to check against the .htaccess rules with each request.

I clean .htaccess file manually once in a while (depends how many entries there are now) and watch the patterns. If it seems to be just a one-time-hit, then I remove the blocking entry. If this seems to be a repetitive bot (with the same, or similar UA string), then I make another, permanent entry to block that bot by UA sting.

I don't understand just how I should handle the redirects.

I don't use redirects. I just deny access (403) to the whole website.