How to block nonsense user agents in .htaccess

bfgbkwxu5sniwhakebif5rgxlfaapynhn

[Wizcrafts]

For the past week or two I have been seeing people download entire pages (html, css scripts and SSI), in one GET, all having random gibberish characters and numbers as their User Agent. They do not act like browsers because they do not download images, google ads, javascript includes and certain other items I embed in my web pages. They take it all in one get, like this obfuscated example:

64.***.***.*** - - [28/Feb/2006:03:31:11 -0500] "GET / HTTP/1.0" 200 34982 "-" "tkprmiugimgckntaook3hxuvvtail"

The user agents are created by a random string generator, and may contain one, two, three, or four (or possibly more) groups of characters. Each group may contain as little as one character, or more than 30. They are comprised of upper and lower case vowels and consonants and numbers. I have even seen non-ascii characters in one string.

Here are some of the actual user agent strings I have extracted from my logs over the past few days:

ftgklkksfcjbxjujgnaw
pivnobwyube5xrmfXycodfvwrkahdmxn5wdtqq
r rcxkjttlqgvaylpsDjnbbcrbw
byn pacabwgrbgjtjotvnfthlddwrn
canogqkv ndtf5ypKkeicastrpck
wsadq sah9pfjrrlmhgremk ditrtIuws
jg8gqw8rheebbr8ats grksib bhu gey

Here is the ruleset I have created by trial and error, for use in an Apache .htaccess file, which traps 100% of the above listed strings and sends them a 403 response.

RewriteCond %{HTTP_USER_AGENT} ^[a-zA-Z0-9]{18,}$ [OR]
RewriteCond %{HTTP_USER_AGENT} (.+)\s+(.+)\s*.?
RewriteCond %{HTTP_USER_AGENT} !(asterias在ot列JNetworkQuality圭ompatible圭rawler圩irefox刖FN\ Link\ Scanner夙oogle多ttp:夷a_archiver危edia妃sie妃sn妃ozilla好utch她pera如layer存earch存lurp存pider宅oyager名ebRingChecker名hois安indows) [NC]
RewriteRule (.*) - [F]

The exclusion list is important to this ruleset, for without it many legitimate user agents could be blocked. I'm sure that there are other ways to block these user agent strings, but this is what I came up with, which works for me. The exclusion list can be updated as needed. If you intend to use this ruleset in your .htaccess be sure to change the broken pipe symbols to solid pipes, or you will get a 500 server error and lock everybody out of your website.

I have been informed by an associate that the following additional rule may help trap these bogus strings as well, but I have not tested it at the time of this Posting:

# no vowels
RewriteCond %{HTTP_USER_AGENT} [b-df-hj-np-tvwxz]{5,} [NC]

Wiz

[edited by: jdMorgan at 4:35 am (utc) on Mar. 2, 2006]

[extras]

Is it always all lowercase?
If so, maybe you can use the fact that only small number of good USER_AGENT start with lowercase.

# Block all USER_AGENT starting with lowercase
RewriteCond %{HTTP_USER_AGENT} ^[a-z]
# Allow msnbot and whatever other one.
RewriteCond %{HTTP_USER_AGENT}!^(msnbot¦whatever)
RewriteRule ^ - [L]

For the RewriteRule, I prefer to serve blank page without any data.
RewriteRule!^/*blank.html /blank.html [L]

It's cheaper for the server (as Apache doesn't have to make error.log entry, and don't need to process ErrorDocument directives).
It saves bandwidth, too. :)

Also, you can use the fact that most of valid browsers start with "Mozilla".

[Wizcrafts]

I have tweaked these conditions a little since I first posted this topic. The user agents do contain some capital letters, in random places. Here is my modified rule, which captures gibberish user agents with even fewer failures:

RewriteCond %{HTTP_USER_AGENT} ^[a-zA-Z0-9]{18,}$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^[a-zA-Z0-9]{18,}\s.+ [OR]
RewriteCond %{HTTP_USER_AGENT} (.+)\s+(.+)\s*.? [NC,OR]
# no vowels in 5 characters
RewriteCond %{HTTP_USER_AGENT} [b-df-hj-np-tvwxz]{5,} [NC]
RewriteCond %{HTTP_USER_AGENT} !(asterias冰lackBerry在ot列JNetworkQuality圭ompatible圭rawler圩irefox刖FN\ Link\ Scanner夙oogle多ttp:夷a_archiver夷Tunes危acintosh危edia妃sie妃sn妃ozilla好utch她pera她s=Mac如layer吊uickTime存earch存lurp存pider宅oyager名ebRingChecker名hois安indows) [NC]
RewriteRule .* - [F]
# Alternate ending:
# RewriteRule (.*) /subdirectory/denied.html [L]
# Where denied.html contains only: <h1>Access Denied</h1>

Wiz