How to Ban Bots which come without http_user_agent info?

It's real problem, I don't know how to set it up with rewrite mod

[Chris_1977]

for example:

216.251.82.99 www.opasoft.com /work/scheduler.php 404 14065 - 2003/03/17/06:58:54 "-"

the browser type is "-", how to ban this type agent using rewrite engine?

[jdMorgan]

Chris1977,

There are several threads here on how to ban by user-agent, including blank ones. However, I would caution you that most visitors with a blank user-agent are not malicious; Rather, they are users visiting from behind corporate firewalls, or using software such as Norton Internet Security or Opera with the user-agent hidden.

I suggest you block problem visitors by IP address, or consider installing a bad-bot trap script - also much discussed here. Try the WebmasterWorld site search at the top of every page for background.

If you decide to go ahead and block blank referrers, you could use

RewriteCond %{HTTP_REFERER} ^$
RewriteRule .* - [F]

but again, you'll be blocking mostly innocent users.

HTH,
Jim

[carfac]

or using software such as Norton Internet Security or Opera with the user-agent hidden

Jim:

I get quite a few US'a that say "Blocked By Norton." I beleive Norton can be configured a few different ways, blocking the UA entirely being one of them. But I think you have to WORK to do that ("Blocked By Norton" seems to be the default)

I mention this because, if it does take work to do, then whoever is doing it KNOWS what they are doing. If one has a "You Are Blocked Because" page, that sort of user will either move on, or re-adjustr Norton...

dave

[jdMorgan]

Dave,

This depends on the version of NIS. I've seen older ones come in primarily blank.

Jim

[bull]

sorry, post was wrong here, please delete my message....

[ncw164x]

Unless you don't want Google to spider your site I would be careful on adding "-" to a ban list, Google has this in the UA

"-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)

[ncw164x]

So does Inktomi

"-" "Mozilla/5.0 (Slurp/cat; [e-mail]; [inktomi.com...]

[wilderness]

ncw,
Things are a bit confused here. At least presently.
The initial inquiry is not a standard FULL log example.
His referer in the example above is opasoft
Jim likely provided an example based on that log type.

For your log types "referrer" would not be used.
Rather UA.
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule ^.*$ - [F]

I been using this quite a while and it only denies either blank or alternative blank UA's. Has no effect on blank referrer field.
We all get way too many of those to restrict.

Don

[ncw164x]

Sorry wilderness I was just posting on the comment

"the browser type is "-", how to ban this type agent using rewrite engine?"

[wilderness]

ncw
No need to apologize to anybody here. We're all here to help each other. :)

I haven't looked at the specific format for log type/definitions in some time. So I'm sorta of going by memory which isn't too good these days.
Especially since sticking with Jim last night ;)
Analog used to have online some very good examples and explanations of the numerous log types. I haven't a clue where to find any similar examples today? Anybody?

The full standard logs have seven (7) data fields.
The last two are primarily our present concern and of course the result we can obtain in the 4th field ;)

There is no field specifically for browser in these log types. Although the UA (last field contains browser ID.)

The 6th field is "referrer"
The seventh field is "user agent"

If somebody has the FULL 7-field logs and desires to restrict based on no referrer than use the example Jim provided.
If somebody has the full 7-field logs and desires to restrict on user agent than use the example I provided.

If you put the two examples side by side? It's obvious one says User Agent and the other says Referrer.
Hope this helps

Don

[carfac]

When I do this ban (and if you want it, I will post it) I block when UA AND referer is blank. They BOTH have to be blank to be banned. I think this is a good comprimise!

Dave