.htaccess vs. no-name UA's

[Tzujan]

Ok, so in a discussion around here (forget where) I learned how to ban a bot if it doesn't have a user agent name. But, it's not working for me. Here's what I'm doing, and I'm confused because I keep getting hits to my domain by user agents with no name:

Options +FollowSymlinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_REFERER} ^-?$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^-?$ [NC,OR]
RewriteRule .* - [F,L]

.....what could I possibly be doing wrong?

[claus]

The rule above bans a UA that has an optional "-" as UA string, so it ought to work if the "-" was missing as well.

I can think of one extra case which might look like an empty string, ie: one or more spaces.

Otherwise, i'm not quite sure if these two cases are strictly the same:

1) An empty UA string
2) No UA string

Perhaps case 1 is interpreted in another way than case 2 by the rule, but somebody else will have to comment upon that, as i'm only guessing here and i might be totally wrong.

[DanA]

It seems that no UA or empty UA make no difference in my logs, but most of the time they are not bots but very often Firefox or Mozilla (+user agent switcher -it has one option by default) users who switch UA when they read it makes a difference when browsing the site.

[jdMorgan]

Tzujan,

Welcome to WebmasterWorld [webmasterworld.com]!

The final RewriteCond in a ruleset must never have an "OR" flag on it...

I agree with DanA that you will have problems if you disallow access to blank user-agents. However, that is your choice. So, remove the [OR] flag from your second rule, and your rule should work. Note that there is also no reason to use the [NC] (No Case) flag, since you are not matching alphabetic characters here, so you can omit that, too. [L] used with [F] is also redundant.

RewriteCond %{HTTP_REFERER} ^-?$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule .* - [F]

For practical use without causing problems for legitimate users, I'd suggest using the following code instead:
# BLOCK *Faked* blank referer -OR- UA
RewriteCond %{HTTP_REFERER} ^-$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^-$
RewriteRule .* - [F]
#
# BLOCK blank referer -AND- UA except for HEAD requests (e.g. AOL cache requests)
RewriteCond %{REQUEST_METHOD} !^HEAD$
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule .* - [F]
[/code]
Jim