homepage Welcome to WebmasterWorld Guest from 54.145.183.169
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
76.74.153.xx
no robots.txt , blank user agent
Bewenched

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4522096 posted 4:40 am on Nov 24, 2012 (gmt 0)

76.74.153.99

no robots.txt , blank user agent

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4522096 posted 9:15 pm on Nov 24, 2012 (gmt 0)

76.74.128.0/17 peer-1 - blocked

Bewenched

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4522096 posted 9:44 pm on Nov 24, 2012 (gmt 0)

Thanks dstiles, blocked them now... not sure what they were up to, hit a few random un-attached pages and left.

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4522096 posted 11:16 pm on Nov 24, 2012 (gmt 0)

Blank UA? Aren't you already blocking those at the gate?

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4522096 posted 1:45 am on Nov 25, 2012 (gmt 0)

@ Bewenched

Try using the Search utility at the top nav list. One of the great features of WW is that most things have been discussed, resolved and archived.

Bewenched

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4522096 posted 5:26 am on Nov 25, 2012 (gmt 0)

I'm not sure how to block blank user agents. I do have .htaccess and block a lot of the bad ones, but how do you block a blank one?

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4522096 posted 6:20 am on Nov 25, 2012 (gmt 0)

One of various ways, this is probably the easiest to understand.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule - [F]

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4522096 posted 10:06 am on Nov 25, 2012 (gmt 0)

Or, if you're already using mod_setenvif, you can add it to the list there:

BrowserMatch ^-?$ go_away

or similar. The form ^-?$ means "the full name from beginning to end is either - or nothing". The - is because blank items usually get changed to - in transit. So your logs say "-" rather than "". (This is also the reason for the mysterious pair of hyphens - - that come between the IP and the date in most log entries ;))

Bewenched

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4522096 posted 6:54 pm on Nov 25, 2012 (gmt 0)

Looking at my logs I've realized i cannot block blank user agents. If you run an ecommerce site the post backs from various payment gateways including google checkout, paypal and others do not give a user agent and they would be blocked... this is bad.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4522096 posted 8:21 pm on Nov 25, 2012 (gmt 0)


Looking at my logs I've realized i cannot block blank user agents. If you run an ecommerce site the post backs from various payment gateways including google checkout, paypal and others do not give a user agent and they would be blocked... this is bad.

Then allow those IP ranges.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteCond %{REMOTE_ADDR} !^66\.249\.[6-9][0-9]\.
RewriteCond %{REMOTE_ADDR} !^203\.208\.[3-6][0-9]\.
RewriteRule - [F]

* The ranges I exemplified are Googlebot, but change as needed.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved