Blocking UA

Is it possible to block the UA "-" "-" in htaccess?

[KonnyQ]

Hi guys ( and girls of course;-) )

I have a question according to this closed thread:

[webmasterworld.com...]

In my htaccess there's a hole list of bad bots blocked via UA as well as image protection via Referer. The problem is, that this blocking is worthless when "they" come in with the UA like "-" "-" .

I've read the above thread regarding this topic and now I just want to know if the following would work:

RewriteCond %{HTTP_USER_AGENT} ^"-"\ "-" [OR]
RewriteCond %{HTTP_USER_AGENT} nextbadbot [NC,OR]
RewriteRule .* - [F]

Question: is the syntax for defining the UA right and could this cause problems that I still don't know about?

Any helping ideas?

Best greets, Konny

[KonnyQ]

After a propper sitesearch and reading a bit more I found this thread:

[webmasterworld.com...]

Damn, what a cool forum this is! Hope one fine day I gonna be able to pay back what I've learned here! ;))

[Andrue]

This link is also a similar discussion on that and would help.
[webmasterworld.com...]

[jdMorgan]

KonnyQ,

The following is from code I posted in one of those cited threads, but cleaned up. Note that using the [NC] (no case) flag with a non-alphabetic pattern is unneccessary, and that using the [L] flag with an [F] flag is redundant. So, cleaning up what I had posted there, we get this:

RewriteCond %{HTTP_REFERER} ^-?$
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule .* - [F]

This blocks requests with blank referrer and user-agent, and also blocks requests where either or both fields actually contain a hyphen. Some troublemakers actually populate those request fields with hyphens, specifically to bypass the blocks set up for blank referrer and user-agent! Since most logging routines show a blank request field as "-", this little "visual trick" was used to bypass the blank-block and confuse the webmaster. But it doesn't work any more if you use the code above. (Thanks to guabito for reporting the "hyphen trick".)

Good reference: Introduction to mod_rewrite [webmasterworld.com]

Jim

[KonnyQ]

Hi jdMorgan,

thanks a lot for the info (and for code cleaning ;)), I haven't heard about this hyphen trick you mentioned.

For Flags I only use the [F] in this rule because there's another rule following, to bann certain "bad" referers, after that I use [R,L].

Ok, I'll watch my logs .. and of course I stick with the reference you posted.

Next thing to do is, send these people and bots to a extra custom page or to hell, but I guess I'll figure that. :)

thanks also Andrue,

Konny

[jdMorgan]

Konny,

If a request matches a rule with an [F] in it, then mod_rewrite immediately returns a 403-Forbidden code, and the request is terminated - so no further rewriting takes place or is needed. If you put an [L] flag in there, such as [F,L], it won't hurt anything, but it also won't change anything, either...

Usually, the only time you don't need or want [L] is when a URL from a single request needs to be rewritten in successive stages in order to reduce overall complexity. Most RewriteRules are exclusive anyway, but using [L] will speed things up if you put your most-used RewriteRules first.

Just some comments to clarify... :)

Jim

[KonnyQ]

jdMorgan,

thanks again for the clarification, now my script is working fine!

By the way: another short note regarding my first post:

No doubt, the line RewriteCond %{HTTP_USER_AGENT} ^"-"\ "-" [OR] is completely nonsense!

My head just get's messed too much sometimes and this was one of the results.

Just to tell, not realy important as my problem isn't one anymore. ;)

Konny

[KonnyQ]

Back again, everything is working fine but I still can't stop asking (stupid?) questions.

After reading a few times that certain SE come along with a empty UA and/or referer to check if a site is cloaking, I'm now wondering if this is still actual and if the "big ones" like google and altavista are still doing this kind of checks? If so, could my htaccess cause trouble like being banned in those search engines?

Hope I'll get another answer .. ;)

Konny