Whois "-" "-"?

How do I block "-" "-" from my Robots.txt file?

[jrota]

I get this logged all the time
and am starting to worry that it's bogging down my system:

IP - - [date] "Get /path/directory/filename HTTP/1.0" 200 2200 "-" "-"

Like it's suggesting:

User-Agent "-"
Referer: "-"

And how do I block this animal in my Robots.txt file?

Thx,

Jim Rota

[jdMorgan]

jrota,

You can't block it using robots.txt - it may not even be a robot. This could be any user-agent browser, spider, downloading tool, etc. that is blocking HTTP_REFERER and is using a blank User-agent string.

You can block this kind of access using mod_rewrite on Apache servers, and scripts and other tricks on Microsoft IIS and other servers, but it's usually better to try to block this kind of access using a specific IP address or IP range. For example, users of Norton Internet Security can set NIS to block referer and User-agent. - in fact, that may be the default high-security setting.

Be careful blocking anything - and do it in the most exact way possible, especially if your income depends on your site(s). I try to strike a balance between effectiveness (at blocking spammers) and selectivity (not blocking legitimate users).

Do a site search here on WebmasterWorld for "blocking user agent IP UA" and you'll likely turn up several threads to provide more info.

Hope this helps,
Jim

[Scooter24]

I did a site search on "blocking user agent IP UA" and got only five results, all of which unrelated to the issue on how to block a nameless "-" user agent.

How would a blocking line in .htaccess look like? Something like

SetEnvIfNoCase User-Agent <something> getout
....
Deny from env=getout
</Limit>

What is the correct syntax for the <something> part?

[wilderness]

This will work

RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule ^.*$ [F]

[gsx]

User agent and referrer are often blocked by firewalls.

[jdMorgan]

Scooter24,

Here are some example SetEnvIf directives for blocking by IP address:

SetEnvif Remote_Addr ^1\.2\.3\.4$ keep_out #block IP address 1.2.3.4
SetEnvif Remote_Addr ^1\.2\.3\..* keep_out #block IP addresses 1.2.3.xx

To block using mod_rewrite, try:

RewriteCond %{REMOTE_ADDR} ^127\.0\.0\.1$
RewriteRule .* - [F,L]

Check out the Apache documentation [httpd.apache.org] on mod_rewrite [httpd.apache.org] and mod_setenv [httpd.apache.org] - Good info!

Jim

[carfac]

Hi:

I think this is a bit better at blocking a black UA AND referer:

RewriteCond %{HTTP_REFERER} ^-?$ [NC]
RewriteCond %{HTTP_USER_AGENT} ^-?$ [NC]
RewriteRule ^.*$ [F,L]

And please correct me if I am wrong!

dave

[edit] not sure if you see it, but there HAS to be at least one white space between the "}" and the "^", and the "$" and the "[" on the first two lines, and between Rule and "^" and the "$" and the "[" on the second line [/edit]

[jdMorgan]

A couple minor changes:

RewriteCond %{HTTP_REFERER} ^-?$ [NC]
RewriteCond %{HTTP_USER_AGENT} ^-?$ [NC]
RewriteRule .* - [F,L]

Jim

[carfac]

Jim:

Whoops- you are right!

dave

PS, Jim- I was gonna sticky you about this block- what do you think of it... or anyone else who cares to comment about blocking anyone with blank UA AND a blank Referer...

I know it may block some people with Nortoin, but they will know why, I would think (Doesn't Norton default to "Blocked By Nortoon" and you have to make it send nothing...)

dave

[jdMorgan]

Caution is indicated for commercial sites... I actually use a variant of this for only certain special cases, and redirect such blocks to an information page with "help" on Norton Internet Security settings, and other possible causes. Otherwise, you might get:

Worker: Is there some guy in the IS department named Norton?
Boss: No, why?
Worker: Because he's blocking my computer!

Jim

[Finder]

I've been trying to figure this out myself. I get lots of hits from IPs with no referrer and no UA, and they usually just get one page (usually root) and that's it.

I am assuming they are bots, but are they harmful? What are they looking for?