Nasty robot with insane user agent - help

Insane user agent

[Moosenskjoeld]

Allo,

I have been hit today with a nasty robot. I don't know what to do. Some specifics:

UA: '*+`\xdf?=)(/&%$\xa7\!
IP: different each time
Host: ditto
Requests files at an inhuman rate

Each time it hits, it has a different IP and different 'provider'. Regular, popular ISP services.

If I were to ban by IP, I'd feel like playing a flipper game from the 80s. If I were to deny by provider, I'd end up banning thousands of normal viewers.

As you see, the UA is insane. It's way beyond my knowledge what to do with such an attacker....

Do you have any hints as to what could be done?

best regards,

M.

[wilderness]

Jim,
Is there a rewrite to include nonalpha and/or numeral multiple digits?

Something like?

If begins with XXX and non-alpa or non-numeric, than, whatever?

[T_Rex]

That's why I have a security concern about the bad-bot-script. Is your .htaccess file size rapidly going up to slow down your page delivery?

[wilderness]

rapidly going up to slow down your page delivery

hardly!

[Span]

Different IPs, different hosts, but the same UA?
If so, RewriteCond %{HTTP_USER_AGENT} xdf will match/catch it.

[Moosenskjoeld]

Unfortunately, this doesn't do it. I have no idea why. I usually do that with substrings, and it always works.

What I think is that the UA is in Unicode, or some encoding, and what the server logs present is not the true UA.

Try searching for that phrase to see what Google brings up.

Thank you for help so far,

M.
(helpless in Chicago)

[isitreal]

That's why I have a security concern about the bad-bot-script.

the bad bot script only triggers when its page is requested, look at the get requests on your log files, are they requesting / or index page, or not even that? If so, it's not really spidering your site, it's more of an attack or a probe, one of my sites has been hit by this for weeks now, but it's not a spider, it's something else, about 50 mB a day of probes, my hoster got rid of about 2/3's of the attack but we haven't been able to get rid of the rest, I've been too lazy to follow up on it since it's not really affecting performance at this point. But looking at the log files showed me one thing clearly, this is not a standard get request, and so will have nothing to do with the bad bot script's function.

The bad spiders are crawling your site, this exploit is doing something else, it's probably not even requesting pages, it's trying to break into the server if I remember right, but double check your log files. Mine look like this, IP's are random as far as I can tell

82.80.252.221 - - [24/Jul/2004:15:39:48 -0400] "CONNECT 64.59.134.8:25 HTTP/1.0" 200 2317 "-" "-"
82.80.252.221 - - [24/Jul/2004:15:40:09 -0400] "CONNECT 64.59.134.8:25 HTTP/1.0" 200 2317 "-" "-"
81.23.250.198 - - [24/Jul/2004:15:40:12 -0400] "\x04\x01" 200 2317 "-" "-"
81.23.250.198 - - [24/Jul/2004:15:40:40 -0400] "\x04\x01" 200 2317 "-" "-"

[Span]

Oh.. well because of the weird string I thought it was better to test it with wannabrowser before posting and I was actually served a 403.

[Moosenskjoeld]

Perhaps this?

RewriteCond %{HTTP_USER_AGENT}!^[A-Z]+$ [OR]

would do? What do you think?

M.

[RonPK]

RewriteCond %{HTTP_USER_AGENT} !^[A-Z]+$

That might work; all regular UA strings I know have some capital character.

RewriteCond %{HTTP_USER_AGENT} \\xdf

should match the bad bot.

[jdMorgan]

Can we clear this up a bit?

Is the user-agent always this fixed string: '*+`\xdf?=)(/&%$\xa7\!

If so, Span's RewriteCond should work fine. If you see a user-agent string in your log files, you can ban it using what you see.

The full regex pattern would be: ^\'\*\+\\xdf\?=\)\(/&\%\$\\xa7\\\!$

isitreal, you can block by testing %{HTTP_METHOD}. If it isn't GET, HEAD, OPTIONS, POST, PROPFIND, or TRACE, reject it.

Jim

[isitreal]

Thanks jd, I'll give that a try and see how it goes.