Botnets

How to recognise & block

[keyplyr]

Botnets have become an increasing threat to webmasters, made up of compromised accounts from various sources: DSL, Cable Broadband, ISPs, Telecoms, Mobile, Server Farms, Cloud, Colo, Corporate and Private lines... the list grows.

It's easy to feel powerless over methods to block these attacks. They usually have valid, normal request headers and human looking UA strings.

I was one who though it was too much to deal with and for a long time took no action, but I've started adding these IP address to my block list a few months ago.

At my sites I have noticed this behavior from compromised IP addresses:
• One particular page is requested w/ no other supporting files, no other requests.
• The above hit may also be accompanied by a hit for a php file or other dynamic file type.
• Consecutive page requests (HTML files only) each with a different UA string.
• Consecutive directory file(s) requests.
• Various requests for wp (Wordpress) files
• Various requests for login or admin files
• The above hits come from ISP type ranges, but also come from server farms.

Sometimes prior to the scrapes/attacks, the compromised accounts are tested (YMMV.)
• One particular page is requested w/ no other supporting files, no other requests. This is to 1.) check to see if the compromised IP addresses is still valid, and 2.) evaluate the victim's server response.
• Different bad actors may request completely different pages for much the same reason.

- Then the BIG ATTACK comes -

My botnet IP list (in addition to server farm IP ranges) grows each day, but I have successfully blocked many attacks. A few new IPs do make it through, and of course get added to this list.

[dstiles]

Generally /22 or /21 - depends on the appearance, severity and stupidity of hit and often a quick scan for open ports on three or four IPs in the range: but that only gets done if I'm in a giving mood. :)

I don't think you're correct about only being /22 but in any case ranges get moved to new owners from time to time.

I agree, though that RIPE is very short of IP ranges. The newly released ones have mostly been taken by ARIN. :) And then US companies move in and take over blocks of RIPE IPs for crummy servers! And I get questioned when I ask for a small /27 ! :(

[Pfui]

aristotle, re 'self-referrals': Been sighting/fighting those "uri=ref" hits since 2008: [webmasterworld.com...]

Year in and year out, most hail from major server farms and/or notorious countries, so if you're not 403'ing those, do.

[aristotle]

Phui
Thanks for your reply. I read the thread you referenced, but I didn't see any explanation for the purpose or motive of whoever is behind this activity.

For example, in my case there are about 200 new requests everyday, all of which are blocked with a 403 forbidden. But no matter how many are blocked, new ones continue to come in everyday at about the same rate of 200 per day. Also, nearly all of the IPs are U.S. locations.

Let me make some points:

1. As I said earlier, this level of activity (200 requests per day) is far too feeble to be a serious "attack" against my site. It doesn't knock it off-line or interfere with its operation at all. It is much too weak for that. Nobody would be dumb enough to waste their botnet resouces on something that's so ineffective. So it isn't an "attack".

2. It obviously isn't scraping, or probing or attempted hacking either, since it always requests the same static html file. At the rate of 200 requests per day, this will eventually add up to tens (or hundreds) of thousands of requests, all apparently from a different IP, all for the same file. How could anyone think that this is scraping, probing, or hacking.

3. Earlier in this thread I mentioned my theory that this is testing that takes place as new devices are infected with the malware and added to the botnet.

So my basic question, which as far as I can tell nobody has answered, is WHAT IS THE PURPOSE OR MOTIVE OF WHOEVER IS CONTROLING THIS BOTNET? IN OTHER WORDS, WHAT ARE THEY TRYING TO ACCOMPLISH?
Thanks again

[wilderness]

WHAT IS THE PURPOSE OR MOTIVE OF WHOEVER IS CONTROLING THIS BOTNET? IN OTHER WORDS, WHAT ARE THEY TRYING TO ACCOMPLISH?

aristotle,
There is rarely a clear explanation of what these malicious bots are doing and what their end-use for the data is.
There are many possibilities, however a webmaster generally never gets an valid answer.

The crawls are certainly NOT desired, beneficial, and in many instances violation of accepted protocol and TOS.

[aristotle]

wilderness wrote:
There are many possibilities

Are you talking about botnets, which is what this thread is supposed to be about? If you are talking about botnets, then please list some of these many possibilities, because I would really like to see them.
P.S. For the particular case I described, I've already ruled out things like scraping, probing for vulnerabilities, hack attempts, etc.

[wilderness]

wilderness wrote:
There are many possibilities

Are you talking about botnets, which is what this thread is supposed to be about?

aristotle,
I beginning to believe that you just like f..king with me ;)

Compromised machines (whether Botnets or anything else) never offer a clear definition of their purpose.

As one example:
Search the WW forums for FunWebProducts, and you see the controversy and non-clarity of this product.

I'm here to tell you that this product was on compromised machines, and in many instances pages were crawled by the product on its own.

Believe it or don't believe it.

So what's the difference between a Botnet and another bot (malicious, non-major-SE or otherwise) that offers ZERO benefit to your website (s)?

[aristotle]

wilderness
I'm sorry you feel that way, but the simple truth is that I'm just trying to understand the motive of the individual or group that's behind the activity that I'm seeing on my website. You said that there are many possibilities to consider, and so I asked you to list some them in the hope that it would help me. Now you seem to be saying that I'll never understand it, and so I shouldn't even try, and that therefore there's no point in discussing it with me

[wilderness]

I'm not sure what you mean by "attacks"
So two questions:
<snip>

Sometimes prior to the scrapes/attacks, the compromised accounts are tested (YMMV.)
• One particular page is requested w/ no other supporting files, no other requests. This is to 1.) check to see if the compromised IP addresses is still valid, and 2.) evaluate the victim's server response.
• Different bad actors may request completely different pages for much the same reason.

keyplyr
Thanks for your reply. I was merely trying to understand what you meant by the word "attack", but if you don't want to tell me, that's your privilage.

f..king with people!

keyplr clearing defined attack in his opening submission.

That your unable to comprehend is either his issue (nor mine).

You were already told ("but if you don't want to tell me, that's your privilage") bears no repetition.

[lucy24]

I don't think you're correct about only being /22

I can't remember where and when exactly I read the announcement. I've got a nebulous idea it was in the latter part of 2012, though I may be wrong about this detail. I do keep track of IP changes by country every few months. In RIPE, absolutely nothing is happening in IPv4 except the bit-by-bit doling out of 185 in /22 slivers. /22 i.e. 1024 unique addresses is also the total allocation of the entire nation of North Korea. Brr.

Definitely not where I first read it, but worth it for the picture :)
[engadget.com...]

There's also this:
[ripe.net...]
first question

[dstiles]

aristotle - don't knock people who say there are many possible reasons for a botnet to operate the way it does. As wilderness said, there is no clear reason.

If someone is continually hitting one of your pages there ARE many reasons and it's quite possible no one here could hit on the true reason. For example, a continual hit on a single page MAY be trying to find a chink through which it could slide a virus, ane be hitting that one page because the botnet operator THINKS it is a popular page that could get a good return on investment if successful. Or it MAY be someone thinks the content is, as Americans say, "awesome" and they are trying to scrape the content for any one of a multitude of nefarious purposes from a semi-legit "search engine" effort to implanting it into another site as a come-on, probably with a virus payload added.

There are, as we'va all said here, any number of reasons and we certainly do not know even a fratcion of them. Think up a scenario, it's probable someone is already probing.

If you want an overview of potential nastiness you could always join a hackers forum. You can be be very discreet about it, although in the end they will probably figure you out. :)

[dstiles]

Lucy - ok, you win. :)

[lucy24]

:p

Don, I think Aristotle was asking the deeper and fuzzier question: Not "What are their intentions w/r/t my server", but "What's in it for them?"

And if you find yourself wondering why a robot based in the Seychelles would be interested in your widgets, just imagine how _I_ feel :)

[wilderness]

"What's in it for them?"

403s, more 403s, and still more 403s. . . .