Welcome to WebmasterWorld Guest from 54.166.46.226

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

76.74.153.xx

no robots.txt , blank user agent

   
4:40 am on Nov 24, 2012 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



76.74.153.99

no robots.txt , blank user agent
9:15 pm on Nov 24, 2012 (gmt 0)

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 5+ Year Member



76.74.128.0/17 peer-1 - blocked
9:44 pm on Nov 24, 2012 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Thanks dstiles, blocked them now... not sure what they were up to, hit a few random un-attached pages and left.
11:16 pm on Nov 24, 2012 (gmt 0)

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



Blank UA? Aren't you already blocking those at the gate?
1:45 am on Nov 25, 2012 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



@ Bewenched

Try using the Search utility at the top nav list. One of the great features of WW is that most things have been discussed, resolved and archived.
5:26 am on Nov 25, 2012 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



I'm not sure how to block blank user agents. I do have .htaccess and block a lot of the bad ones, but how do you block a blank one?
6:20 am on Nov 25, 2012 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



One of various ways, this is probably the easiest to understand.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule - [F]
10:06 am on Nov 25, 2012 (gmt 0)

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



Or, if you're already using mod_setenvif, you can add it to the list there:

BrowserMatch ^-?$ go_away

or similar. The form ^-?$ means "the full name from beginning to end is either - or nothing". The - is because blank items usually get changed to - in transit. So your logs say "-" rather than "". (This is also the reason for the mysterious pair of hyphens - - that come between the IP and the date in most log entries ;))
6:54 pm on Nov 25, 2012 (gmt 0)

WebmasterWorld Senior Member 5+ Year Member



Looking at my logs I've realized i cannot block blank user agents. If you run an ecommerce site the post backs from various payment gateways including google checkout, paypal and others do not give a user agent and they would be blocked... this is bad.
8:21 pm on Nov 25, 2012 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month




Looking at my logs I've realized i cannot block blank user agents. If you run an ecommerce site the post backs from various payment gateways including google checkout, paypal and others do not give a user agent and they would be blocked... this is bad.

Then allow those IP ranges.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteCond %{REMOTE_ADDR} !^66\.249\.[6-9][0-9]\.
RewriteCond %{REMOTE_ADDR} !^203\.208\.[3-6][0-9]\.
RewriteRule - [F]

* The ranges I exemplified are Googlebot, but change as needed.