homepage Welcome to WebmasterWorld Guest from 54.197.215.146
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
76.74.153.xx
no robots.txt , blank user agent
Bewenched




msg:4522098
 4:40 am on Nov 24, 2012 (gmt 0)

76.74.153.99

no robots.txt , blank user agent

 

dstiles




msg:4522240
 9:15 pm on Nov 24, 2012 (gmt 0)

76.74.128.0/17 peer-1 - blocked

Bewenched




msg:4522244
 9:44 pm on Nov 24, 2012 (gmt 0)

Thanks dstiles, blocked them now... not sure what they were up to, hit a few random un-attached pages and left.

lucy24




msg:4522258
 11:16 pm on Nov 24, 2012 (gmt 0)

Blank UA? Aren't you already blocking those at the gate?

keyplyr




msg:4522270
 1:45 am on Nov 25, 2012 (gmt 0)

@ Bewenched

Try using the Search utility at the top nav list. One of the great features of WW is that most things have been discussed, resolved and archived.

Bewenched




msg:4522298
 5:26 am on Nov 25, 2012 (gmt 0)

I'm not sure how to block blank user agents. I do have .htaccess and block a lot of the bad ones, but how do you block a blank one?

keyplyr




msg:4522299
 6:20 am on Nov 25, 2012 (gmt 0)

One of various ways, this is probably the easiest to understand.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule - [F]

lucy24




msg:4522334
 10:06 am on Nov 25, 2012 (gmt 0)

Or, if you're already using mod_setenvif, you can add it to the list there:

BrowserMatch ^-?$ go_away

or similar. The form ^-?$ means "the full name from beginning to end is either - or nothing". The - is because blank items usually get changed to - in transit. So your logs say "-" rather than "". (This is also the reason for the mysterious pair of hyphens - - that come between the IP and the date in most log entries ;))

Bewenched




msg:4522406
 6:54 pm on Nov 25, 2012 (gmt 0)

Looking at my logs I've realized i cannot block blank user agents. If you run an ecommerce site the post backs from various payment gateways including google checkout, paypal and others do not give a user agent and they would be blocked... this is bad.

keyplyr




msg:4522414
 8:21 pm on Nov 25, 2012 (gmt 0)


Looking at my logs I've realized i cannot block blank user agents. If you run an ecommerce site the post backs from various payment gateways including google checkout, paypal and others do not give a user agent and they would be blocked... this is bad.

Then allow those IP ranges.

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteCond %{REMOTE_ADDR} !^66\.249\.[6-9][0-9]\.
RewriteCond %{REMOTE_ADDR} !^203\.208\.[3-6][0-9]\.
RewriteRule - [F]

* The ranges I exemplified are Googlebot, but change as needed.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved