A one-line solution in .htaccess for banning bad user-agents?

[teasers]

Hi there,

Of course there are hundreds of possible names for bad user agents out there but, what about adding "most dangerous words" in one line into .htaccess, and then add some other known ones like DA or HTTrack, etc?

Adding:

RewriteCond %{HTTP_USER_AGENT} (copier¦collect¦download¦express¦extract¦grab¦mail¦mass¦offline) [NC]

will restirict all user-agents which include those names, whether they're at the beginning or at the end and even synonyms like:

downloader (cause we have the bad word download there)
eeemail (we have the word mail)
email#*$!xxer
copyemail
thebestsitegrabber (grab)

I think that would help when new versions of those programs come online, and even new programs (or user-agents, let's say) if they do contain those words. Not a complete solution of course, but what are your thoughts?

[jdMorgan]

Sure, this technique will work and it wil reduce the number of variants that you have to test explicitly.

The thing you have to watch out for is that it is remotely possible that some new harmless user-agent will appear that is blocked by this code, so it's not a good solution to implement and then forget about. For example, this would be of some concern if you were doing work for a client rather than for your own site (which you could closely monitor for a few months, at least).

Be careful of the "mail" user-agents, most are "bad", but some may be "good", depending on your site. Some, such as EmailProtect, seem to be email spam filtering services that check the message's claimed originating domain to be sure that it exists. For something like that, a separate block is needed, rather than a blanket policy:

# Block e-mail harvesters but not emailprotect
RewriteCond %{HTTP_USER_AGENT} E-?mail [NC]
RewriteCond %{HTTP_USER_AGENT}!EmailProtect
RewriteRule .* - [F]

That said, you might consider adding
attach¦copy¦entry¦extract¦Ninja¦Stripper¦Sucker¦Whacker
to your list, and I'm sure there are more.

Jim

[dcrombie]

I think you'll find that none of the user agents that you're targetting with those RegExps are still circulating.

The current generation of spambots (at least those that can be identified from the user agent) includes:

- Holiday Shopping, Inc.
- Just a Browser
- NASA Search 1.0

and the tail end of the 'random' strings:

- etdiwgwqhftxchndjwictekjp
- Bgibe ldxnljBqoong hB
- jmuphl iGhdgwjpulginwloxs7vqbbgwj

Not a sign of 'EmailSiphon' or similar names for months...