This. Is. Nasty.

[lucy24]

I don't normally notice 403s. Out of sight, out of mind. I noticed this one, because yesterday's log file was ten times the size of normal. Maybe twenty; it's a weekend.

Here's why.

One:

192.3.150.164 - - [24/May/2014:15:14:34 -0700] "GET / HTTP/1.1" 403 1787 "http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CCgQFjAA&url=http%3A%2F%2Fexample.com%2F&ei=yBmBU82LAoecqAbVsICAAw&usg=AFQjCNHM5ePue9VzOlY1b5T9QYGikEvxyg&bvm=bv.67720277,d.b2k&cad=rja" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0" 
192.3.150.164 - - [24/May/2014:15:14:34 -0700] "GET /boilerplate/errorstyles.css HTTP/1.1" 200 2954 "http://example.com/" "{same}" 
192.3.150.164 - - [24/May/2014:15:14:34 -0700] "GET /piwik/piwik.js HTTP/1.1" 403 1786 "http://example.com/" "{same}" 
192.3.150.164 - - [24/May/2014:15:14:35 -0700] "GET /favicon.ico HTTP/1.1" 200 661 "-" "{same}" 

In other words, exactly what you'd get from a wrongly blocked human. (The IP is ColoCrossing.) I have holes poked for css and favicon for just this reason.

Two: (excerpt)

192.3.150.164 - - [24/May/2014:15:14:46 -0700] "GET /hovercraft/ HTTP/1.1" 403 1786 "http://example.com/" "{same}" 
192.3.150.164 - - [24/May/2014:15:14:46 -0700] "GET /piwik/piwik.js HTTP/1.1" 403 1786 "http://example.com/hovercraft/" "{same}" 
192.3.150.164 - - [24/May/2014:15:14:47 -0700] "GET /fonts/ HTTP/1.1" 403 1786 "http://example.com/hovercraft/" "{same}" 
192.3.150.164 - - [24/May/2014:15:14:48 -0700] "GET /piwik/piwik.js HTTP/1.1" 403 1786 "http://example.com/fonts/" "{same}" 
192.3.150.164 - - [24/May/2014:15:14:49 -0700] "GET /fun/ HTTP/1.1" 403 1786 "http://example.com/fonts/" "{same}" 

Verisimilitude begins to wear thin, as it requests every single page linked from error documents. (The 403 page says something like "If you keep getting bounced back to this page, it means the server thinks you are a robot.")

Three:
Tiring of this, it proceeds to:

192.3.150.164 - - [24/May/2014:15:16:10 -0700] "HEAD / HTTP/1.1" 403 164 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [24/May/2014:15:16:11 -0700] "GET /dildo/ HTTP/1.1" 403 3320 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [24/May/2014:15:16:11 -0700] "HEAD /index/ HTTP/1.1" 403 164 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [24/May/2014:15:16:11 -0700] "HEAD /download/ HTTP/1.1" 403 164 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [24/May/2014:15:16:11 -0700] "HEAD /images/ HTTP/1.1" 403 164 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [24/May/2014:15:16:12 -0700] "HEAD /2006/ HTTP/1.1" 403 164 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [24/May/2014:15:16:12 -0700] "HEAD /crack/ HTTP/1.1" 403 164 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [24/May/2014:15:16:12 -0700] "HEAD /serial/ HTTP/1.1" 403 164 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [24/May/2014:15:16:11 -0700] "GET / HTTP/1.1" 403 3320 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [24/May/2014:15:16:12 -0700] "HEAD /warez/ HTTP/1.1" 403 164 "-" "RAPE RAPE RAPE RAPE" 
<snip>
192.3.150.164 - - [25/May/2014:10:19:17 -0700] "GET /dildo HTTP/1.1" 403 1731 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [25/May/2014:10:19:18 -0700] "GET /stats/ HTTP/1.1" 401 606 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [25/May/2014:10:34:07 -0700] "GET /dildo HTTP/1.1" 403 1731 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [25/May/2014:10:34:07 -0700] "GET /stats/ HTTP/1.1" 401 606 "-" "RAPE RAPE RAPE RAPE" 

and so on for a total of 12,604 requests so far.

Note the timestamp. They are still going on. Wonder if the host can do a temporary firewall?

Note the UA. That is the actual, literal UA string, not an editorial emendation.

[keyplyr]

Just my 2 cents...

1.) Nothing good comes from ColoCrossing. I only allow it to get robots.txt and of course my custom 403 page.

2.) I was told early on to never link any other documents to a 403 page. You can see why :)

As an aside... I always check who I'm blocking. I don't actually run an IP check on each and every 403, but I do manually drill through my raw logs every day, usually every couple hours, and make sure I'm not blocking the wrong people. I can recognize a lot of the ranges by sight and I do look a lot of them up, just not *all* of them.

As far as the RAPE UA goes, there's an awful lot of browser extensions/plugins that allow UA spoofing. The reason for this specific UA is anyone's guess. Passing a psychological test is not required to own a computer... yet.

[edited by: keyplyr at 2:28 am (utc) on May 26, 2014]

[not2easy]

Maybe feed them a unique error doc or redirect to a 410? Just trying to think what i would try, probably won't make it stop. Nasty is right.

[lucy24]

There's a complicating factor which I didn't immediately realize: one of the two pages it keeps requesting is /stats/. Since the /stats/ directory isn't physically located inside my userspace (shared hosting, so it's all done in some central place), all the htaccess lockouts in the world won't help. Instead this request gets a 401 error, leading the server to request (internally) the host-default 401 page-- which I don't happen to have, because why would I.

For the other request-- the one that, ahem, doesn't exist-- I've temporarily lifted the IP block, and replaced it with a redirect to 127.0.0.1. Sending them off to contemplate their navels used to work wonders with some Ukrainian robots. Here unfortunately it doesn't seem to have any effect whatsoever. For the past nine hours they've been eating 401, 301, 401, 301 just as avidly as they used to eat 401, 403, 401, 403. The one comfort is that a 301 response is only 488 bytes-- with no associated internal requests-- while the 403 was 1731 plus requests for various internal files.

Ugh, ugh, ugh. Most of the time I'm happy to blame robotic behaviors on stupidity alone. This one's malice.

[aristotle]

Is this still going on?

Also, In a case like this, does the source see the defensive actions you take against it, and if so, does it normally ignore them or not?

[lucy24]

They stopped as suddenly as they started, about 24 hours ago (I make that 45 hours after they started).

There is no way to know whether they acted on the redirect to 127.0.0.1. And all I know about the 401 is that the server sent it out. They didn't see the 401 page, because it doesn't exist; I have to assume the server sent out the 403 page instead. (Since the 401 page doesn't exist, it isn't covered by the FilesMatch and preliminary RewriteRule exemptions covering most error documents.)

I have no idea what's up with, ugh, /dildo, but it seems safe to guess that they know most sites have a /stats/ directory, and they're hoping to find it unprotected. (Query: What can a malign robot do with analog stats if it's able to access them?)

[aristotle]

Glad to hear that it stopped.
But apparently my other question wasn't clear, so if I may ask again in another way, could your defensive actions have had anything to do with it stopping?

One reason I ask is because there's a situation on one of my sites where bots from a botnet have been downloading the same two pages for months. New UAs and IPs have been appearing everyday, and I'm worried that it's a preparatory buildup for an eventual DDOS attack. With your help in another thread, I set up some defensive measures earlier, but that hasn't stopped them from appearing. But it did have an effect, because if my measures block them from downloading the first page, then they don't even try to download the second page.

In other words, they react to being blocked at the first page. But I don't know what enables them to react in this way, and would like to ask if anyone has an explanation.

[tangor]

With scammers, scrapers and scum they will hammer for a bit until someone views the log results to see if there's a chink in the target. They might also view the target act in defense. That's no joy for them as they want in easy peasy to do their evil. So, they will set sights on the next IP/URL of desire.

As for the filenames requested... each leads to other types of spam/scam they can do... and reveals the biz of the site targeted. Some might be higher value then others.

The Stats directory most times is used to produce referer (sic) spam... though the big SEs are starting of fight back on that (and we webmasters are hurting in not getting the best referer info as we used to get. It all changes, day in day out, but the attempt, and the methods used, remain the same.

Lastly, if some of the these requested directories are actually VISIBLE to these miscreants, they just might find a way into to spam the site or do other uglies.

[lucy24]

could your defensive actions have had anything to do with it stopping?

It's always possible. I mentioned earlier that some Ukrainian robots seem to go away faster if you respond with a 127.0.0.1 redirect instead of a flat 403. Some people have also tried a redirect to the offender's own IP (this can be done easily in htaccess with no php-script involvement). Use caution with this one, because it might be a proxy; if so, you should really be redirecting to the "real" IP (sometimes visible in an X-Forwarded-For header).

A redirect isn't as viscerally satisfying as a 403. But it does create a little less work for the server-- this is assuming you're on shared hosting and can't do a firewall-- because you're sending out nothing except a response header. No content.

[tangor]

I do send something. It looks like this: yln ... yeah, that's he right number of fingers... they get it. :)

I don't recommend it, but redirect to a USA gov spy agency is also fun to think about. :)

[keyplyr]

@Lucy, well your friend came round my place. Endless 403'd requests for all things a hacker's mind could ever wish for. It's been going on for hours.

192.3.150.164 - - [30/May/2014:00:19:55 -0700] "HEAD /SourceCode HTTP/1.1" 403 307 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [30/May/2014:00:19:55 -0700] "HEAD /cerberus HTTP/1.1" 403 307 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [30/May/2014:00:19:55 -0700] "HEAD /Win HTTP/1.1" 403 307 "-" "RAPE RAPE RAPE RAPE" 
192.3.150.164 - - [30/May/2014:00:19:55 -0700] "HEAD /Crack HTTP/1.1" 403 307 "-" "RAPE RAPE RAPE RAPE" 

I sent abuse reports up the chain of command to: [e-mail], [e-mail], [e-mail], [e-mail].

[lucy24]

About 1000 403'd requests

They must be getting tired. My final total was almost 13,000 requests, the vast majority in an initial 25-minute blast. The last 500 or so were the same two names over & over in alternation. For a while earlier they were asking for names in the form
/1234/
(looks like anywhere from 3-7 digits, some in yyyy-mm form or similar, never in numerical order). I didn't exactly fine-tooth-comb the list, but I cut it out of logs and saved as a separate file. It's one hell of a shopping list. Aggregate total-- I had the text editor count for me-- 12499 403's, 268 401's (those were the /stats/ requests) and 131 301's.

:: business with calculator ::

Oh, right, and two 200's for the non-page requests at the beginning. They never hit fast enough to trigger a 500-class error. Onn my server it's, I think, 30 simultaneous connections, a misbehavior I associate with Bezeq from a few years back.

:: further business with calculator ::

13000 requests in 25 minutes is, what, about 8 per second? Enough to avoid triggering errors if they're extremely methodical-- especially since almost all were HEAD alone. Like I said: not stupidity but malice.

Edit: You must have fined-tuned your post while I had the tab open, hence the phantom quote.

[keyplyr]

Ahhhh... phantom quotes, my favorites!

behavior was the same... big blast for about an hour, then little bursts for the next couple hours, mostly just one hit every 15 minutes with same garbage, but with me it never went for deep stats mining, just a couple requests for stats here and there.

When all was done, there were about 8k. (I have a line counter feature in my text editor but it maxes out at a certain point.)

Anyway, no sign of them today. Guess they got tired of RAPEing.

[derfmann]

What i do is monitor request per client/minute and per ip/minute ... if a threshold is reached i slow the response down ... only for that requests ... the more requests come the slower i answer ... normal users wont see that, only the bot.