Blank User agent string

Blank User agent string query

[baiwan]

I've been going through my server logs which I hadn't looked at for quite a while.

I've looked up all the bots both here and elsewhere on the web, to determine which to block - this forum in particular has been very useful.

I've noticed what looks like bot activity - 600+ page fetches with little time between them; e.g:

92.28.117.### 23/25/2010 2:34:11
92.28.117.### 23/25/2010 2:34:13
92.28.117.### 23/25/2010 2:34:15
92.28.117.### 23/25/2010 2:34:16
92.28.117.### 23/25/2010 2:34:18
92.28.117.### 23/25/2010 2:34:21
92.28.117.### 23/25/2010 2:34:23
92.28.117.### 23/25/2010 2:34:25
92.28.117.### 23/25/2010 2:34:27
92.28.117.### 23/25/2010 2:34:29
92.28.117.### 23/25/2010 2:34:33
92.28.117.### 23/25/2010 2:34:36
92.28.117.### 23/25/2010 2:34:37

What's the best/easiest way to deal with this kind of activity?

[edited by: Ocean10000 at 12:05 am (utc) on Apr 24, 2010]
[edit reason] Masked IP's [/edit]

[Ocean10000]

Welcomed baiwan

There are different methods of blocking problem like this. But they depend on what web server you are using to serve the web pages with, and possibly of what extras are installed with the web server.

I simply block all requests without a valid User-Agent, which stops a lot of pest before they can become problems. I do this on site I control with Asp.Net, but there are various ways to do it.


But for other details the Ip range belongs to Opal-Net, which is a major internet provider. So it would not be wise to block them as a whole.

% Information related to '92.28.0.0/15AS13285'
route: 92.28.0.0/15
descr: Opal-Net Autonomous System
origin: AS13285
mnt-by: OPAL-MNT
source: RIPE # Filtered

inetnum: 92.24.0.0 - 92.29.255.255
[1545076736] - [1545469951]
netname: OPAL-DSL
descr: Opal Telecom DSL
country: GB
admin-c: PM58-RIPE
tech-c: PM58-RIPE
status: ASSIGNED PA
mnt-by: OPAL-MNT
mnt-lower: OPAL-MNT
mnt-routes: OPAL-MNT
source: RIPE # Filtered
address: Opal Telecommunications Plc
address: Northbank Industrial Estate
address: Irlam
address: Manchester
address: M44 5BL
address: United Kingdom

[edited by: incrediBILL at 8:21 pm (utc) on Apr 24, 2010]
[edit reason] Removed specifics [/edit]

[baiwan]

I'm on shared hosting - Linux/Apache/Mysql/PHP etc.

I was concerned that there might be legitimate agents that failed to provide a UA string, but looking more closely at my logs, it's probably not the case.

About 3% of page requests are from UA's that don't provide a UA string.

Can they be blocked in .htaccess? If not I'll trap them in my PHP.

Thanks!

[wilderness]

was concerned that there might be legitimate agents that failed to provide a UA string,

NONE

Can they be blocked in .htaccess?

"User agent" +blank [google.com]

RewriteEngine on (comment;if not already on;)
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule .* - [F]