76.74.153.xx

no robots.txt , blank user agent

[Bewenched]

76.74.153.99

no robots.txt , blank user agent

[dstiles]

76.74.128.0/17 peer-1 - blocked

[Bewenched]

Thanks dstiles, blocked them now... not sure what they were up to, hit a few random un-attached pages and left.

[lucy24]

Blank UA? Aren't you already blocking those at the gate?

[keyplyr]

@ Bewenched

Try using the Search utility at the top nav list. One of the great features of WW is that most things have been discussed, resolved and archived.

[Bewenched]

I'm not sure how to block blank user agents. I do have .htaccess and block a lot of the bad ones, but how do you block a blank one?

[keyplyr]

One of various ways, this is probably the easiest to understand.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^-?$ 
RewriteRule - [F]

[lucy24]

Or, if you're already using mod_setenvif, you can add it to the list there:

BrowserMatch ^-?$ go_away

or similar. The form ^-?$ means "the full name from beginning to end is either - or nothing". The - is because blank items usually get changed to - in transit. So your logs say "-" rather than "". (This is also the reason for the mysterious pair of hyphens - - that come between the IP and the date in most log entries ;))

[Bewenched]

Looking at my logs I've realized i cannot block blank user agents. If you run an ecommerce site the post backs from various payment gateways including google checkout, paypal and others do not give a user agent and they would be blocked... this is bad.

[keyplyr]

Looking at my logs I've realized i cannot block blank user agents. If you run an ecommerce site the post backs from various payment gateways including google checkout, paypal and others do not give a user agent and they would be blocked... this is bad.

Then allow those IP ranges.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteCond %{REMOTE_ADDR} !^66\.249\.[6-9][0-9]\.
RewriteCond %{REMOTE_ADDR} !^203\.208\.[3-6][0-9]\.
RewriteRule - [F]

* The ranges I exemplified are Googlebot, but change as needed.